AbuseIPDB Integration module

[marcopolo]

I'm facing an issue with the spider detection code. The purpose of this code is to detect spiders/crawlers, if enabled will bypass the api call and creates a log log if enabled as well. However, I've noticed that the log file is not being created even though I can see spiders accessing my site.

I suspect there might be an issue with the spider detection logic or the file logging process. I would greatly appreciate it if someone with more knowledge and experience could take a look at the code snippet and help me identify the problem.

            // Skip API call for known spiders if enabled
                if (isset($spider_flag) && $spider_flag === true && $spider_allow == 'true') {

                    // Check if logging is enabled for allowed spiders
                        $log_file_name_spiders = 'abuseipdb_spiders_' . date('Y_m') . '.log';
                        $log_file_path_spiders = $log_file_path . $log_file_name_spiders;
                        $log_message = date('Y-m-d H:i:s') . ' IP address ' . $ip . ' Spider - Score: ' . $abuseScore . PHP_EOL;

                    if ($spider_log_enabled == 'true') {            
                        file_put_contents($log_file_path_spiders, $log_message, FILE_APPEND);
                    }

                return 0; // Return 0 score for spiders or whatever default value you want
                } 

I have already verified that the variables ($spider_flag, $spider_allow, $spider_log_enabled, etc.) are correctly set and the paths for the log files are valid.

Could someone please review the code and provide insights into why the log file is not being created? Any suggestions, improvements, or alternative approaches to spider detection in Zen Cart are also welcome.

Thank you in advance for your assistance!

[lat9]

Being lazy and not downloading the plugin, if $log_file_path is set to DIR_FS_LOGS, then the $log_file_name_spiders should have a leading '/' since DIR_FS_LOGS doesn't end in that character.

[marcopolo]

Being lazy and not downloading the plugin, if $log_file_path is set to DIR_FS_LOGS, then the $log_file_name_spiders should have a leading '/' since DIR_FS_LOGS doesn't end in that character.

That is not the issue, the module creates a few different logs which are all working fine. It is just this spider detection one not being created. The log file setting is set within the admin setting of the module, like so for my install: log/

Here is the admin setting:
('Log File Path', 'ABUSEIPDB_LOG_FILE_PATH', 'logs/', 'The path to the directory where log files are stored.', $cgi, now(), 45, NULL, NULL),

[lat9]

The code currently will run that section of code (logging the spider detection) only when the ABUSEIPDB_SPIDER_ALLOW setting's value is true and allow spiders to continue with the checks when false; was that the intent?

[marcopolo]

The code currently will run that section of code (logging the spider detection) only when the ABUSEIPDB_SPIDER_ALLOW setting's value is true and allow spiders to continue with the checks when false; was that the intent?

Yes that is the intent, if you allow spiders then this will allow you to avoid an api call check for them, if you do not allow them then the module checks their ip like any other users ip for an abuse score and will allow or block them based on your set threshold.

[marcopolo]

For testing purposes, I've hardcoded my IP address ('xxx.xxx.xxx.xxx') and manually set $spider_flag to true. The intention is to simulate a scenario where a known spider accesses the website.

Here's the relevant code snippet:

// Hardcoding the IP for testing
$ip = 'xxx.xxx.xxx.xxx';
$spider_flag = ($ip == 'xxx.xxx.xxx.xxx') ? true : $spider_flag; // Set $spider_flag to true if the IP is xxx.xxx.xxx.xxx

// Skip API call for known spiders if enabled
if (isset($spider_flag) && $spider_flag === true && $spider_allow == 'true') {

    // Check if logging is enabled for allowed spiders
    $log_file_name_spiders = 'abuseipdb_spiders_' . date('Y_m') . '.log';
    $log_file_path_spiders = $log_file_path . $log_file_name_spiders;
    $log_message = date('Y-m-d H:i:s') . ' IP address ' . $ip . ' is identified as a Spider. AbuseIPDB API check was bypassed.' . PHP_EOL;

    if ($spider_log_enabled == 'true') {            
        file_put_contents($log_file_path_spiders, $log_message, FILE_APPEND);
    }

    return 0; // Return 0 score for spiders or whatever default value you want
} 

The test setup, wherein I've hardcoded my IP and set $spider_flag to true, results in successful logging - a log file gets created as expected. This suggests that the issue is unlikely with the logging mechanism itself.

Moreover, while monitoring the 'Who is Online' section, I've noticed several spiders accessing the site. However, the corresponding spider log file has not been created, leading me to believe that the spider detection part of the code is not working as expected.

To summarize, while my test setup with hardcoded values demonstrates that the logging functionality is intact, it appears that under normal site operation, spiders are not being correctly detected and logged.

It brings us to an interesting dilemma. If 'Who is Online' is accurately identifying spiders, then it appears that my spider detection code is not functioning correctly. However, if my code is working properly and there haven't been any spider visits that Zen Cart would identify as spiders, then 'Who is Online' might be giving false positives. Either way, there seems to be a discrepancy that warrants further investigation.

I'd appreciate any insights or suggestions on this matter.

[marcopolo]

This may be an alternative way detect spiders:

            // Skip API call for known spiders if enabled
            $user_agent = $_SERVER['HTTP_USER_AGENT'];
            $spiders_file = DIR_WS_INCLUDES . 'spiders.txt';
            $spiders = file($spiders_file, FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES);

            foreach ($spiders as $spider) {
                if (strpos($user_agent, $spider) !== false) {
                $spider_flag = true;
                break;
                }
            }
                if ($spider_flag && $spider_allow == 'true') {

                    // Check if logging is enabled for allowed spiders
                        $log_file_name_spiders = 'abuseipdb_spiders_' . date('Y_m') . '.log';
                        $log_file_path_spiders = $log_file_path . $log_file_name_spiders;
                        $log_message = date('Y-m-d H:i:s') . ' IP address ' . $ip . ' is identified as a Spider. AbuseIPDB API check was bypassed.' . PHP_EOL;

                    if ($spider_log_enabled == 'true') {            
                        file_put_contents($log_file_path_spiders, $log_message, FILE_APPEND);
                    }

                return 0; // Return 0 score for spiders or whatever default value you want
                } 

[lat9]

Noting that the $spider_flag is set IFF SESSION_FORCE_COOKIE_USE != 'True' and SESSION_BLOCK_SPIDERS == 'True' (from the zc158 /includes/init_includes/init_sessions.php lines 70-84).

Don't know if that helps ...