Known spiders are definitively prevented from creating sessions
As you are not using the page_not_found redirect, you could change in the observer class from
Code:
// Do not execute the check for the 'page_not_found' page
if ($current_page_base == 'page_not_found') {
return;
}
to
Code:
// Do not execute the check for known spiders
if (isset($spider_flag) && $spider_flag === true) {
return;
}
to disable the whole thing for known spiders
I'm not using this on a live site at the moment so cannot really test if its working