AbuseIPDB Integration module

[marcopolo]

I'm facing an issue with the spider detection code. The purpose of this code is to detect spiders/crawlers, if enabled will bypass the api call and creates a log log if enabled as well. However, I've noticed that the log file is not being created even though I can see spiders accessing my site.

I suspect there might be an issue with the spider detection logic or the file logging process. I would greatly appreciate it if someone with more knowledge and experience could take a look at the code snippet and help me identify the problem.

            // Skip API call for known spiders if enabled
                if (isset($spider_flag) && $spider_flag === true && $spider_allow == 'true') {

                    // Check if logging is enabled for allowed spiders
                        $log_file_name_spiders = 'abuseipdb_spiders_' . date('Y_m') . '.log';
                        $log_file_path_spiders = $log_file_path . $log_file_name_spiders;
                        $log_message = date('Y-m-d H:i:s') . ' IP address ' . $ip . ' Spider - Score: ' . $abuseScore . PHP_EOL;

                    if ($spider_log_enabled == 'true') {            
                        file_put_contents($log_file_path_spiders, $log_message, FILE_APPEND);
                    }

                return 0; // Return 0 score for spiders or whatever default value you want
                } 

I have already verified that the variables ($spider_flag, $spider_allow, $spider_log_enabled, etc.) are correctly set and the paths for the log files are valid.

Could someone please review the code and provide insights into why the log file is not being created? Any suggestions, improvements, or alternative approaches to spider detection in Zen Cart are also welcome.

Thank you in advance for your assistance!

[lat9]

Being lazy and not downloading the plugin, if $log_file_path is set to DIR_FS_LOGS, then the $log_file_name_spiders should have a leading '/' since DIR_FS_LOGS doesn't end in that character.

[marcopolo]

Being lazy and not downloading the plugin, if $log_file_path is set to DIR_FS_LOGS, then the $log_file_name_spiders should have a leading '/' since DIR_FS_LOGS doesn't end in that character.

That is not the issue, the module creates a few different logs which are all working fine. It is just this spider detection one not being created. The log file setting is set within the admin setting of the module, like so for my install: log/

Here is the admin setting:
('Log File Path', 'ABUSEIPDB_LOG_FILE_PATH', 'logs/', 'The path to the directory where log files are stored.', $cgi, now(), 45, NULL, NULL),

[lat9]

The code currently will run that section of code (logging the spider detection) only when the ABUSEIPDB_SPIDER_ALLOW setting's value is true and allow spiders to continue with the checks when false; was that the intent?

[marcopolo]

The code currently will run that section of code (logging the spider detection) only when the ABUSEIPDB_SPIDER_ALLOW setting's value is true and allow spiders to continue with the checks when false; was that the intent?

Yes that is the intent, if you allow spiders then this will allow you to avoid an api call check for them, if you do not allow them then the module checks their ip like any other users ip for an abuse score and will allow or block them based on your set threshold.

[marcopolo]

For testing purposes, I've hardcoded my IP address ('xxx.xxx.xxx.xxx') and manually set $spider_flag to true. The intention is to simulate a scenario where a known spider accesses the website.

Here's the relevant code snippet:

// Hardcoding the IP for testing
$ip = 'xxx.xxx.xxx.xxx';
$spider_flag = ($ip == 'xxx.xxx.xxx.xxx') ? true : $spider_flag; // Set $spider_flag to true if the IP is xxx.xxx.xxx.xxx

// Skip API call for known spiders if enabled
if (isset($spider_flag) && $spider_flag === true && $spider_allow == 'true') {

    // Check if logging is enabled for allowed spiders
    $log_file_name_spiders = 'abuseipdb_spiders_' . date('Y_m') . '.log';
    $log_file_path_spiders = $log_file_path . $log_file_name_spiders;
    $log_message = date('Y-m-d H:i:s') . ' IP address ' . $ip . ' is identified as a Spider. AbuseIPDB API check was bypassed.' . PHP_EOL;

    if ($spider_log_enabled == 'true') {            
        file_put_contents($log_file_path_spiders, $log_message, FILE_APPEND);
    }

    return 0; // Return 0 score for spiders or whatever default value you want
} 

The test setup, wherein I've hardcoded my IP and set $spider_flag to true, results in successful logging - a log file gets created as expected. This suggests that the issue is unlikely with the logging mechanism itself.

Moreover, while monitoring the 'Who is Online' section, I've noticed several spiders accessing the site. However, the corresponding spider log file has not been created, leading me to believe that the spider detection part of the code is not working as expected.

To summarize, while my test setup with hardcoded values demonstrates that the logging functionality is intact, it appears that under normal site operation, spiders are not being correctly detected and logged.

It brings us to an interesting dilemma. If 'Who is Online' is accurately identifying spiders, then it appears that my spider detection code is not functioning correctly. However, if my code is working properly and there haven't been any spider visits that Zen Cart would identify as spiders, then 'Who is Online' might be giving false positives. Either way, there seems to be a discrepancy that warrants further investigation.

I'd appreciate any insights or suggestions on this matter.

[marcopolo]

This may be an alternative way detect spiders:

            // Skip API call for known spiders if enabled
            $user_agent = $_SERVER['HTTP_USER_AGENT'];
            $spiders_file = DIR_WS_INCLUDES . 'spiders.txt';
            $spiders = file($spiders_file, FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES);

            foreach ($spiders as $spider) {
                if (strpos($user_agent, $spider) !== false) {
                $spider_flag = true;
                break;
                }
            }
                if ($spider_flag && $spider_allow == 'true') {

                    // Check if logging is enabled for allowed spiders
                        $log_file_name_spiders = 'abuseipdb_spiders_' . date('Y_m') . '.log';
                        $log_file_path_spiders = $log_file_path . $log_file_name_spiders;
                        $log_message = date('Y-m-d H:i:s') . ' IP address ' . $ip . ' is identified as a Spider. AbuseIPDB API check was bypassed.' . PHP_EOL;

                    if ($spider_log_enabled == 'true') {            
                        file_put_contents($log_file_path_spiders, $log_message, FILE_APPEND);
                    }

                return 0; // Return 0 score for spiders or whatever default value you want
                } 

[lat9]

Noting that the $spider_flag is set IFF SESSION_FORCE_COOKIE_USE != 'True' and SESSION_BLOCK_SPIDERS == 'True' (from the zc158 /includes/init_includes/init_sessions.php lines 70-84).

Don't know if that helps ...

[marcopolo]

Noting that the $spider_flag is set IFF SESSION_FORCE_COOKIE_USE != 'True' and SESSION_BLOCK_SPIDERS == 'True' (from the zc158 /includes/init_includes/init_sessions.php lines 70-84).

Don't know if that helps ...

I stumbled on that during my debugging and my settings are inline with that:

Based on my settings (SESSION_FORCE_COOKIE_USE = false and SESSION_BLOCK_SPIDERS = true), the spider detection code should work as intended, but it is not.

[marcopolo]

I have been testing the spider detection functionality in Zen Cart to recognize my user agent string as a spider. However, I'm experiencing unexpected behavior and the spider detection does not seem to be working as intended.

Steps taken:

Modified the spiders.txt file to include the necessary entry for the user agent string I want to identify as a spider. in my case my user agent is:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36

so I added: gecko to the spiders.txt

Added debug logging statements in the code to check the values of the spider blocking flag and spider flag variables.

// Log the spider flag value for debugging
if (defined('SESSION_BLOCK_SPIDERS')) {
    error_log('AbuseIPDB Check - Spider Blocking Enabled: ' . (SESSION_BLOCK_SPIDERS ? 'true' : 'false'));
} else {
    error_log('AbuseIPDB Check - Spider Blocking status is not defined.');
}

if (isset($spider_flag)) {
    error_log('AbuseIPDB Check - Spider Flag: ' . ($spider_flag ? 'true' : 'false'));
} else {
    error_log('AbuseIPDB Check - Spider Flag is not set.');
} 

Expected the debug logs to be created and indicate that the spider blocking flag is enabled and the spider flag is set to true.

However, the debug logs were NOT created when the user agent string matched the entry in spiders.txt, but they were created when I removed the entry.

Possible considerations:

Is there additional logic or conditions in the spider detection mechanism that I might be missing?
Are there other files or functions involved in the spider detection process that I should review?
Could there be any conflicting factors or checks affecting the spider detection outcome?

I kindly request the assistance of the Zen Cart community in understanding and resolving this issue. Any insights, suggestions, or guidance would be greatly appreciated.

Thank you for your time and assistance.