AbuseIPDB Integration module

[marcopolo]

The code currently will run that section of code (logging the spider detection) only when the ABUSEIPDB_SPIDER_ALLOW setting's value is true and allow spiders to continue with the checks when false; was that the intent?

Yes that is the intent, if you allow spiders then this will allow you to avoid an api call check for them, if you do not allow them then the module checks their ip like any other users ip for an abuse score and will allow or block them based on your set threshold.

[marcopolo]

For testing purposes, I've hardcoded my IP address ('xxx.xxx.xxx.xxx') and manually set $spider_flag to true. The intention is to simulate a scenario where a known spider accesses the website.

Here's the relevant code snippet:

// Hardcoding the IP for testing
$ip = 'xxx.xxx.xxx.xxx';
$spider_flag = ($ip == 'xxx.xxx.xxx.xxx') ? true : $spider_flag; // Set $spider_flag to true if the IP is xxx.xxx.xxx.xxx

// Skip API call for known spiders if enabled
if (isset($spider_flag) && $spider_flag === true && $spider_allow == 'true') {

    // Check if logging is enabled for allowed spiders
    $log_file_name_spiders = 'abuseipdb_spiders_' . date('Y_m') . '.log';
    $log_file_path_spiders = $log_file_path . $log_file_name_spiders;
    $log_message = date('Y-m-d H:i:s') . ' IP address ' . $ip . ' is identified as a Spider. AbuseIPDB API check was bypassed.' . PHP_EOL;

    if ($spider_log_enabled == 'true') {            
        file_put_contents($log_file_path_spiders, $log_message, FILE_APPEND);
    }

    return 0; // Return 0 score for spiders or whatever default value you want
} 

The test setup, wherein I've hardcoded my IP and set $spider_flag to true, results in successful logging - a log file gets created as expected. This suggests that the issue is unlikely with the logging mechanism itself.

Moreover, while monitoring the 'Who is Online' section, I've noticed several spiders accessing the site. However, the corresponding spider log file has not been created, leading me to believe that the spider detection part of the code is not working as expected.

To summarize, while my test setup with hardcoded values demonstrates that the logging functionality is intact, it appears that under normal site operation, spiders are not being correctly detected and logged.

It brings us to an interesting dilemma. If 'Who is Online' is accurately identifying spiders, then it appears that my spider detection code is not functioning correctly. However, if my code is working properly and there haven't been any spider visits that Zen Cart would identify as spiders, then 'Who is Online' might be giving false positives. Either way, there seems to be a discrepancy that warrants further investigation.

I'd appreciate any insights or suggestions on this matter.

[marcopolo]

This may be an alternative way detect spiders:

            // Skip API call for known spiders if enabled
            $user_agent = $_SERVER['HTTP_USER_AGENT'];
            $spiders_file = DIR_WS_INCLUDES . 'spiders.txt';
            $spiders = file($spiders_file, FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES);

            foreach ($spiders as $spider) {
                if (strpos($user_agent, $spider) !== false) {
                $spider_flag = true;
                break;
                }
            }
                if ($spider_flag && $spider_allow == 'true') {

                    // Check if logging is enabled for allowed spiders
                        $log_file_name_spiders = 'abuseipdb_spiders_' . date('Y_m') . '.log';
                        $log_file_path_spiders = $log_file_path . $log_file_name_spiders;
                        $log_message = date('Y-m-d H:i:s') . ' IP address ' . $ip . ' is identified as a Spider. AbuseIPDB API check was bypassed.' . PHP_EOL;

                    if ($spider_log_enabled == 'true') {            
                        file_put_contents($log_file_path_spiders, $log_message, FILE_APPEND);
                    }

                return 0; // Return 0 score for spiders or whatever default value you want
                } 

[lat9]

Noting that the $spider_flag is set IFF SESSION_FORCE_COOKIE_USE != 'True' and SESSION_BLOCK_SPIDERS == 'True' (from the zc158 /includes/init_includes/init_sessions.php lines 70-84).

Don't know if that helps ...

[marcopolo]

Noting that the $spider_flag is set IFF SESSION_FORCE_COOKIE_USE != 'True' and SESSION_BLOCK_SPIDERS == 'True' (from the zc158 /includes/init_includes/init_sessions.php lines 70-84).

Don't know if that helps ...

I stumbled on that during my debugging and my settings are inline with that:

Based on my settings (SESSION_FORCE_COOKIE_USE = false and SESSION_BLOCK_SPIDERS = true), the spider detection code should work as intended, but it is not.

[marcopolo]

I have been testing the spider detection functionality in Zen Cart to recognize my user agent string as a spider. However, I'm experiencing unexpected behavior and the spider detection does not seem to be working as intended.

Steps taken:

Modified the spiders.txt file to include the necessary entry for the user agent string I want to identify as a spider. in my case my user agent is:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36

so I added: gecko to the spiders.txt

Added debug logging statements in the code to check the values of the spider blocking flag and spider flag variables.

// Log the spider flag value for debugging
if (defined('SESSION_BLOCK_SPIDERS')) {
    error_log('AbuseIPDB Check - Spider Blocking Enabled: ' . (SESSION_BLOCK_SPIDERS ? 'true' : 'false'));
} else {
    error_log('AbuseIPDB Check - Spider Blocking status is not defined.');
}

if (isset($spider_flag)) {
    error_log('AbuseIPDB Check - Spider Flag: ' . ($spider_flag ? 'true' : 'false'));
} else {
    error_log('AbuseIPDB Check - Spider Flag is not set.');
} 

Expected the debug logs to be created and indicate that the spider blocking flag is enabled and the spider flag is set to true.

However, the debug logs were NOT created when the user agent string matched the entry in spiders.txt, but they were created when I removed the entry.

Possible considerations:

Is there additional logic or conditions in the spider detection mechanism that I might be missing?
Are there other files or functions involved in the spider detection process that I should review?
Could there be any conflicting factors or checks affecting the spider detection outcome?

I kindly request the assistance of the Zen Cart community in understanding and resolving this issue. Any insights, suggestions, or guidance would be greatly appreciated.

Thank you for your time and assistance.

[johnjlarge]

Just a thought.

Could it be that the spiders detection/do not allow sessions for spiders code happens further down in the code stack, so the abuseipdb plugin does its api lookup/redirect before the useragent is detected as a spider from the spiders.txt file?

[marcopolo]

Just a thought.

Could it be that the spiders detection/do not allow sessions for spiders code happens further down in the code stack, so the abuseipdb plugin does its api lookup/redirect before the useragent is detected as a spider from the spiders.txt file?

I checked that I think it's good from the program flow I referenced:
https://docs.zen-cart.com/dev/code/program_flow/

Set up and start session if valid session is above where this module comes in which is: NOTIFY_HTML_HEAD_START (the /includes/templates/common/html_header.php) unless I'm missing something. I tried going further down the list which did not resolve the issue.