AbuseIPDB Integration module

lat9:

Being lazy and not downloading the plugin, if $log_file_path is set to DIR_FS_LOGS, then the $log_file_name_spiders should have a leading '/' since DIR_FS_LOGS doesn't end in that character.

That is not the issue, the module creates a few different logs which are all working fine. It is just this spider detection one not being created. The log file setting is set within the admin setting of the module, like so for my install: log/

Here is the admin setting:
('Log File Path', 'ABUSEIPDB_LOG_FILE_PATH', 'logs/', 'The path to the directory where log files are stored.', $cgi, now(), 45, NULL, NULL),

The code currently will run that section of code (logging the spider detection) only when the ABUSEIPDB_SPIDER_ALLOW setting's value is true and allow spiders to continue with the checks when false; was that the intent?

lat9:

The code currently will run that section of code (logging the spider detection) only when the ABUSEIPDB_SPIDER_ALLOW setting's value is true and allow spiders to continue with the checks when false; was that the intent?

Yes that is the intent, if you allow spiders then this will allow you to avoid an api call check for them, if you do not allow them then the module checks their ip like any other users ip for an abuse score and will allow or block them based on your set threshold.

For testing purposes, I've hardcoded my IP address ('xxx.xxx.xxx.xxx') and manually set $spider_flag to true. The intention is to simulate a scenario where a known spider accesses the website.

Here's the relevant code snippet:

// Hardcoding the IP for testing
$ip = 'xxx.xxx.xxx.xxx';
$spider_flag = ($ip == 'xxx.xxx.xxx.xxx') ? true : $spider_flag; // Set $spider_flag to true if the IP is xxx.xxx.xxx.xxx

// Skip API call for known spiders if enabled
if (isset($spider_flag) && $spider_flag === true && $spider_allow == 'true') {

    // Check if logging is enabled for allowed spiders
    $log_file_name_spiders = 'abuseipdb_spiders_' . date('Y_m') . '.log';
    $log_file_path_spiders = $log_file_path . $log_file_name_spiders;
	$log_message = date('Y-m-d H:i:s') . ' IP address ' . $ip . ' is identified as a Spider. AbuseIPDB API check was bypassed.' . PHP_EOL;

    if ($spider_log_enabled == 'true') {            
        file_put_contents($log_file_path_spiders, $log_message, FILE_APPEND);
    }

    return 0; // Return 0 score for spiders or whatever default value you want
}

The test setup, wherein I've hardcoded my IP and set $spider_flag to true, results in successful logging - a log file gets created as expected. This suggests that the issue is unlikely with the logging mechanism itself.

Moreover, while monitoring the 'Who is Online' section, I've noticed several spiders accessing the site. However, the corresponding spider log file has not been created, leading me to believe that the spider detection part of the code is not working as expected.

To summarize, while my test setup with hardcoded values demonstrates that the logging functionality is intact, it appears that under normal site operation, spiders are not being correctly detected and logged.

It brings us to an interesting dilemma. If 'Who is Online' is accurately identifying spiders, then it appears that my spider detection code is not functioning correctly. However, if my code is working properly and there haven't been any spider visits that Zen Cart would identify as spiders, then 'Who is Online' might be giving false positives. Either way, there seems to be a discrepancy that warrants further investigation.

I'd appreciate any insights or suggestions on this matter.

This may be an alternative way detect spiders:

			// Skip API call for known spiders if enabled
			$user_agent = $_SERVER['HTTP_USER_AGENT'];
			$spiders_file = DIR_WS_INCLUDES . 'spiders.txt';
			$spiders = file($spiders_file, FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES);

			foreach ($spiders as $spider) {
				if (strpos($user_agent, $spider) !== false) {
				$spider_flag = true;
				break;
				}
			}
				if ($spider_flag && $spider_allow == 'true') {

					// Check if logging is enabled for allowed spiders
						$log_file_name_spiders = 'abuseipdb_spiders_' . date('Y_m') . '.log';
						$log_file_path_spiders = $log_file_path . $log_file_name_spiders;
						$log_message = date('Y-m-d H:i:s') . ' IP address ' . $ip . ' is identified as a Spider. AbuseIPDB API check was bypassed.' . PHP_EOL;

					if ($spider_log_enabled == 'true') {			
						file_put_contents($log_file_path_spiders, $log_message, FILE_APPEND);
					}

				return 0; // Return 0 score for spiders or whatever default value you want
				}

Noting that the $spider_flag is set IFF SESSION_FORCE_COOKIE_USE != 'True' and SESSION_BLOCK_SPIDERS == 'True' (from the zc158 /includes/init_includes/init_sessions.php lines 70-84).

Don't know if that helps ...

lat9:

Noting that the $spider_flag is set IFF SESSION_FORCE_COOKIE_USE != 'True' and SESSION_BLOCK_SPIDERS == 'True' (from the zc158 /includes/init_includes/init_sessions.php lines 70-84).

Don't know if that helps ...

I stumbled on that during my debugging and my settings are inline with that:

Based on my settings (SESSION_FORCE_COOKIE_USE = false and SESSION_BLOCK_SPIDERS = true), the spider detection code should work as intended, but it is not.

I have been testing the spider detection functionality in Zen Cart to recognize my user agent string as a spider. However, I'm experiencing unexpected behavior and the spider detection does not seem to be working as intended.

Steps taken:

Modified the spiders.txt file to include the necessary entry for the user agent string I want to identify as a spider. in my case my user agent is:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36

so I added: gecko to the spiders.txt

Added debug logging statements in the code to check the values of the spider blocking flag and spider flag variables.

// Log the spider flag value for debugging
if (defined('SESSION_BLOCK_SPIDERS')) {
    error_log('AbuseIPDB Check - Spider Blocking Enabled: ' . (SESSION_BLOCK_SPIDERS ? 'true' : 'false'));
} else {
    error_log('AbuseIPDB Check - Spider Blocking status is not defined.');
}

if (isset($spider_flag)) {
    error_log('AbuseIPDB Check - Spider Flag: ' . ($spider_flag ? 'true' : 'false'));
} else {
    error_log('AbuseIPDB Check - Spider Flag is not set.');
}

Expected the debug logs to be created and indicate that the spider blocking flag is enabled and the spider flag is set to true.

However, the debug logs were NOT created when the user agent string matched the entry in spiders.txt, but they were created when I removed the entry.

Possible considerations:

Is there additional logic or conditions in the spider detection mechanism that I might be missing?
Are there other files or functions involved in the spider detection process that I should review?
Could there be any conflicting factors or checks affecting the spider detection outcome?

I kindly request the assistance of the Zen Cart community in understanding and resolving this issue. Any insights, suggestions, or guidance would be greatly appreciated.

Thank you for your time and assistance.

Just a thought.

Could it be that the spiders detection/do not allow sessions for spiders code happens further down in the code stack, so the abuseipdb plugin does its api lookup/redirect before the useragent is detected as a spider from the spiders.txt file?

johnjlarge:

Just a thought.

Could it be that the spiders detection/do not allow sessions for spiders code happens further down in the code stack, so the abuseipdb plugin does its api lookup/redirect before the useragent is detected as a spider from the spiders.txt file?

I checked that I think it's good from the program flow I referenced:
https://docs.zen-cart.com/dev/code/program_flow/

Set up and start session if valid session is above where this module comes in which is: NOTIFY_HTML_HEAD_START (the /includes/templates/common/html_header.php) unless I'm missing something. I tried going further down the list which did not resolve the issue.

Recap and Solution I Implemented in v2.0.7 for the Meantime:

Problem:
During testing, I faced difficulties with the spider detection feature in Zen Cart. Specifically, the utilization of the $spider_flag variable did not produce the expected results, particularly in generating logs after detecting a spider using:

if (isset($spider_flag) && $spider_flag === true)

Solution Implemented:
To resolve this issue, I decided to clone the spider detection code from Zen Cart and integrate it directly into the module functions file. This allowed me to incorporate the spider detection functionality without relying on Zen Cart's $spider_flag variable. As a result, the spider detection now works as intended within my custom code.

function checkSpiderFlag() {
    if (defined('SESSION_BLOCK_SPIDERS')) {
        $user_agent_abuseipdb = '';
        if (isset($_SERVER['HTTP_USER_AGENT'])) {
            $user_agent_abuseipdb = strtolower($_SERVER['HTTP_USER_AGENT']);
        }
        $spider_flag_abuseipdb = false;
        if (!empty($user_agent_abuseipdb)) {
            $spiders_abuseipdb = file(DIR_WS_INCLUDES . 'spiders.txt');
            for ($i = 0, $n = sizeof($spiders_abuseipdb); $i < $n; $i++) {
                if (!empty($spiders_abuseipdb[$i]) && substr($spiders_abuseipdb[$i], 0, 4) != '$Id:') {
                    if (is_integer(strpos($user_agent_abuseipdb, trim($spiders_abuseipdb[$i])))) {
                        $spider_flag_abuseipdb = true;
                        break;
                    }
                }
            }
        }
        return $spider_flag_abuseipdb;
    }
    return false;

Although the current solution successfully resolves the logging problem, I am interested in understanding why using the original $spider_flag variable did not produce the expected outcome. If anyone has any insights, suggestions, or guidance that could be a potential resolution that would allow me to revert back to using the $spider_flag in the future, please do let me know!

Thank you for your time and assistance.

marcopolo:

Recap and Solution I Implemented in v2.0.7 for the Meantime:

Problem:
During testing, I faced difficulties with the spider detection feature in Zen Cart. Specifically, the utilization of the $spider_flag variable did not produce the expected results, particularly in generating logs after detecting a spider using:

if (isset($spider_flag) && $spider_flag === true)

> 
> Solution Implemented:
> To resolve this issue, I decided to clone the spider detection code from Zen Cart and integrate it directly into the module functions file. This allowed me to incorporate the spider detection functionality without relying on Zen Cart's $spider_flag variable. As a result, the spider detection now works as intended within my custom code.
> 
> ```php
function checkSpiderFlag() {
    if (defined('SESSION_BLOCK_SPIDERS')) {
        $user_agent_abuseipdb = '';
        if (isset($_SERVER['HTTP_USER_AGENT'])) {
            $user_agent_abuseipdb = strtolower($_SERVER['HTTP_USER_AGENT']);
        }
        $spider_flag_abuseipdb = false;
        if (!empty($user_agent_abuseipdb)) {
            $spiders_abuseipdb = file(DIR_WS_INCLUDES . 'spiders.txt');
            for ($i = 0, $n = sizeof($spiders_abuseipdb); $i < $n; $i++) {
                if (!empty($spiders_abuseipdb[$i]) && substr($spiders_abuseipdb[$i], 0, 4) != '$Id:') {
                    if (is_integer(strpos($user_agent_abuseipdb, trim($spiders_abuseipdb[$i])))) {
                        $spider_flag_abuseipdb = true;
                        break;
                    }
                }
            }
        }
        return $spider_flag_abuseipdb;
    }
    return false;

Although the current solution successfully resolves the logging problem, I am interested in understanding why using the original $spider_flag variable did not produce the expected outcome. If anyone has any insights, suggestions, or guidance that could be a potential resolution that would allow me to revert back to using the $spider_flag in the future, please do let me know!

Thank you for your time and assistance.

Testing here, and it's working really well with the new spiders detection code. Saving a lot of API calls on my site. Also, I've set my threshold at 30% as most of my customers score a 0 threat score, so I'm currently blocking about 10% of hits to the server. I've gone through the logs manually checking bad IPs and there are some real nasty ones in there. This plugin will save a lot of server resources and turn away some of the worst bad bots & crawlers. Invaluable to small shop owners.

Thank you for all the hard work :)

johnjlarge:

Testing here, and it's working really well with the new spiders detection code. Saving a lot of API calls on my site. Also, I've set my threshold at 30% as most of my customers score a 0 threat score, so I'm currently blocking about 10% of hits to the server. I've gone through the logs manually checking bad IPs and there are some real nasty ones in there. This plugin will save a lot of server resources and turn away some of the worst bad bots & crawlers. Invaluable to small shop owners.

Thank you for all the hard work :)

having recently implemented this plugin on a couple of sites, i can truly see its benefit.

it is a GREAT addition to the zen-cart plugin repository.

thanks to @marcopolo for this contribution!

Heya Marco,

I just submitted a PR/Merge request to your Github repository for adding the Contributor Badge onto the admin dashboard to get a higher API daily limits. The changes aren't that serious, but I added a bit to the documentation and upped the version number. I have also submitted, what would now be, 2.1.2 to the ZC Plugins database.

You can view my PR here:
https://github.com/CcMarc/AbuseIPDB/pull/6

Someone on GitHub requested a whos_online feature. I use an enhanced version of whos_online and am currently still on Zen Cart 1.5.5, so I'm not sure about compatibility with newer versions. However, if you'd like to try it, this enhanced version adds the following features directly within the whos_online screen:

• Exclude IPs by AbuseIPDB Threshold: Automatically filter out IPs from the "Who's Online" list based on their AbuseIPDB score exceeding a set threshold.
• Display AbuseIPDB Score: Show the AbuseIPDB score next to each IP address in the "Who's Online" list, indicating the level of potential abuse associated with the IP.
• Shield Icon for Blocked IPs: Display a shield icon for IPs automatically blocked by the module, indicating they have met or exceeded the threshold.
• Ban Icon for Manual Blacklisting: Provide a ban icon that allows administrators to manually blacklist an IP address, adding it to the blacklist file directly from the "Who's Online" interface.

Here's the complete enhanced_whos_online_2.0 plugin with my modifications. There are some image files that go along with the original plugin, in case you're not currently using it and need those:

This is a great plugin. It has certainly reduced the time normally required blocking the script-kiddies.
AbuseIPDB v2.1.2; zc 158a and 2.1.0; PHP 8.3

In admin > ABUSEIPDB Configuration > Redirect URL Option 2 ```
Option 2: 403 Forbidden - If selected, the user will be shown a 403 Forbidden error
message if their IP is found to be abusive. This option provides a more explicit message indicating that the user is forbidden from accessing
the website due to their IP being flagged as abusive.

How can I provide a more explicit message?

The current message is: 
> Access to xxx.xxx.xxx.xxx was denied.
> You don't have authorisation to view this page.

Using the Developer's Tool Kit, you could search for

*You don't have authorisation
*
Then change the file.**

Running ZC 2.1.0

Installed Access Blocker. Struggled to get email verification to work. Ended up using a different email address which then worked. (strange) Anyway, now I'm going over the settings in ZenCart Admin and noticed. (see attached) Basically it wants a user id number from the URL at AbuseIPDB which no longer displays a user ID in the URL. So this seems to be outdated and needs correction or deletion. Or the instructions updated to be useful in finding this information for the field.

Access Blocker and AbuseIPDB are totally different plugins.

Access Blocker uses the ipdata.co site and you need an account from that site for your "ipData Service: API Key" .

AbuseIPDB uses the AbuseIPDB.com site and you need an account from that site for your "AbuseIPDB: API Key"