AbuseIPDB Integration module

That's exactly it. A completely blank page with (in my case) a 403 response. I guess you could set a custom static 403 page via .htaccess to inform the user that their IP is blocked. I couldn't find what was throwing the headers already sent error when using your plugin, so I just worked backwards from the program flow documentation and added the check IP code as early as possible in the code base.

I've hit another issue. Even though I've signed up as a webmaster & put the button on my website to increase API calls to 5000 on abuseipdb.com, I've hit my ip lookup api limit within 3 hours of a 24-hour period. Does the plugin also cache clean IP's or will it simply look up the same clean IP on every page view, meaning each page view equals one api hit?

Yes, the plugin caches both clean and abusive IPs. It only makes an API call if the abuse score is not in the cache or if it has expired.

Instead of adding a new notifier (NOTIFY_HEADER_START) in includes/templates/YOUR_TEMPLATE/common/tpl_header.php you could also use the existing notifier NOTIFY_HTML_HEAD_START in includes/templates/YOUR_TEMPLATE/common/html_header.php and change in the observer class from NOTIFY_HEADER_START to NOTIFY_HTML_HEAD_START.
This way there is no change of existing core files required when installing the plugin.

I'm not sure if the caching is working correctly. I've woken up just now & almost exhausted my API limit for the day at 7:47am

I just enabled debug mode to check the logs & it would seem it's making a fresh call even for the same ip.

From the bottom of the first debug log

[25-May-2023 07:46:22 Europe/London] Checking cache for IP: 54.236.1.11
[25-May-2023 07:46:22 Europe/London] API call made for IP: 54.236.1.11 with score: 63
[25-May-2023 07:46:22 Europe/London] IP 54.236.1.11 blocked from API call

and then the second

[25-May-2023 07:46:39 Europe/London] Checking cache for IP: 54.236.1.11
[25-May-2023 07:46:39 Europe/London] API call made for IP: 54.236.1.11 with score: 63
[25-May-2023 07:46:39 Europe/London] IP 54.236.1.11 blocked from API call

the third

[25-May-2023 07:46:43 Europe/London] Checking cache for IP: 54.236.1.11
[25-May-2023 07:46:44 Europe/London] API call made for IP: 54.236.1.11 with score: 63
[25-May-2023 07:46:44 Europe/London] IP 54.236.1.11 blocked from API call

So it would seem they are all the same IP address but they are looking up each time if I'm not mistaken.

webchills:

Instead of adding a new notifier (NOTIFY_HEADER_START) in includes/templates/YOUR_TEMPLATE/common/tpl_header.php you could also use the existing notifier NOTIFY_HTML_HEAD_START in includes/templates/YOUR_TEMPLATE/common/html_header.php and change in the observer class from NOTIFY_HEADER_START to NOTIFY_HTML_HEAD_START.
This way there is no change of existing core files required when installing the plugin.

Just changed the observer as mentioned here & removed my code from index.php - it does indeed work, blocking works & it also avoids the headers already sent issue I received before. Great observation. Now I just need to solve this caching issue as I'm about to run out of API calls for another day thanks to the same IP's

johnjlarge:

I'm not sure if the caching is working correctly. I've woken up just now & almost exhausted my API limit for the day at 7:47am

I just enabled debug mode to check the logs & it would seem it's making a fresh call even for the same ip.

From the bottom of the first debug log

[25-May-2023 07:46:22 Europe/London] Checking cache for IP: 54.236.1.11
[25-May-2023 07:46:22 Europe/London] API call made for IP: 54.236.1.11 with score: 63
[25-May-2023 07:46:22 Europe/London] IP 54.236.1.11 blocked from API call

and then the second

[25-May-2023 07:46:39 Europe/London] Checking cache for IP: 54.236.1.11
[25-May-2023 07:46:39 Europe/London] API call made for IP: 54.236.1.11 with score: 63
[25-May-2023 07:46:39 Europe/London] IP 54.236.1.11 blocked from API call

the third

[25-May-2023 07:46:43 Europe/London] Checking cache for IP: 54.236.1.11
[25-May-2023 07:46:44 Europe/London] API call made for IP: 54.236.1.11 with score: 63
[25-May-2023 07:46:44 Europe/London] IP 54.236.1.11 blocked from API call

So it would seem they are all the same IP address, but they are looking up each time if I'm not mistaken.

Just another observation on this, could it be that known spiders are prevented from creating a session, so the IP isn't cached in the session cache? I could be way off, but my initial investigation is that this IP is pinterestbot which I have as a robot in my zen cart who's online/admin robot definitions.

johnjlarge:

Just another observation on this, could it be that known spiders are prevented from creating a session, so the IP isn't cached in the session cache? I could be way off, but my initial investigation is that this IP is pinterestbot which I have as a robot in my zen cart who's online/admin robot definitions.

Known spiders are definitively prevented from creating sessions

As you are not using the page_not_found redirect, you could change in the observer class from

// Do not execute the check for the 'page_not_found' page
		if ($current_page_base == 'page_not_found') {
			return;
		}

to

// Do not execute the check for known spiders
		if (isset($spider_flag) && $spider_flag === true) {
			return;
		}

to disable the whole thing for known spiders

I'm not using this on a live site at the moment so cannot really test if its working

webchills:

Known spiders are definitively prevented from creating sessions

As you are not using the page_not_found redirect, you could change in the observer class from

// Do not execute the check for the 'page_not_found' page
if ($current_page_base == 'page_not_found') {
return;
}

> 
> to
> 
> ```
// Do not execute the check for known spiders
		if (isset($spider_flag) && $spider_flag === true) {
			return;
		}

to disable the whole thing for known spiders

I'm not using this on a live site at the moment so cannot really test if its working

I've changed the code as suggested, but I'll have to wait until tomorrow when my api quota resets to see if it works for spiders. If it works that is quite an elegant fix as I manually block bad bot useragents via .htaccess before they load anything from my site.

webchills:

Known spiders are definitively prevented from creating sessions

As you are not using the page_not_found redirect, you could change in the observer class from

// Do not execute the check for the 'page_not_found' page
if ($current_page_base == 'page_not_found') {
return;
}

> 
> to
> 
> ```
// Do not execute the check for known spiders
		if (isset($spider_flag) && $spider_flag === true) {
			return;
		}

to disable the whole thing for known spiders

I'm not using this on a live site at the moment so cannot really test if its working

Great solution this modification in the code introduces an additional check to prevent unnecessary IP abuse checks.

If the current page is a 'page_not_found', the IP abuse check will be skipped.
If the visitor is a known web spider or bot, the IP abuse check will also be skipped.

This reduces unnecessary API calls to AbuseIPDB when traffic comes from known web spiders or bots, which are usually harmless.

// Do not execute the check for the 'page_not_found' page or for known spiders
if ($current_page_base == 'page_not_found' || (isset($spider_flag) && $spider_flag === true)) {
    return;
}

marcopolo:

Great solution this modification in the code introduces an additional check to prevent unnecessary IP abuse checks.

If the current page is a 'page_not_found', the IP abuse check will be skipped.
If the visitor is a known web spider or bot, the IP abuse check will also be skipped.

This reduces unnecessary API calls to AbuseIPDB when traffic comes from known web spiders or bots, which are usually harmless.

// Do not execute the check for the 'page_not_found' page or for known spiders
if ($current_page_base == 'page_not_found' || (isset($spider_flag) && $spider_flag === true)) {
return;
}

I've changed the modification to this to exclude spiders & 404. Just have to wait until tomorrow to check if block caching is now fully working, but on test mode all seems well. This mod seems to be evolving quickly, but it will do wonders to keep the worst offenders away & reduce my manual blocking workload massively.

Thank you all.

Also, as an aside in case anyone else wants to include an error message on their block page, I'm still serving a 403 but now echoing a browser message as follows.

   header('HTTP/1.0 403 Forbidden');
                echo 'You are forbidden! Your IP Address is marked as malicious in the abuseipdb.com database';
                zen_exit();

Any chance we can get a github link to avoid having to add, delete, rinse, repeat?

dbltoe:

Any chance we can get a github link to avoid having to add, delete, rinse, repeat?

Here is the GitHub link: https://github.com/CcMarc/AbuseIPDB.git

I Updated to the latest commit from github, which was made three hours ago, as the API results caching wasn't working on mine. My API calls reset at midnight and by 7am GMT I had hit 3.5k API calls. Checking the calls log, there were multiples for the same IP. The latest commit, saving sessions to database appears to have fixed this. Good work :)

johnjlarge:

I Updated to the latest commit from github, which was made three hours ago, as the API results caching wasn't working on mine. My API calls reset at midnight and by 7am GMT I had hit 3.5k API calls. Checking the calls log, there were multiples for the same IP. The latest commit, saving sessions to database appears to have fixed this. Good work :)

Are you on v2.0? Switched from session caching to database caching for improved performance and reliability.

marcopolo:

Are you on v2.0? Switched from session caching to database caching for improved performance and reliability.

Yes, I'm on version 2.0 and fully up to date as of just a moment ago with the changes on github.

A couple of ideas, in the abuseipdb_api_call_2023_05.log the IPs of spiders which appear in spiders.txt are still showing as blocked, for example

2023-05-26 16:51:52 IP address 54.236.1.11 API call. Score: 63
(this is pinterest bot, which is still allowed to browse the site, but perhaps we could avoid adding and spider sessions to the api log)

Also, maybe beyond the scope of this plugin, but perhaps we could stop blocked IPs from showing in whos online?

Blocking is working really well, just thinking how the plugin could evolve. Perhaps even excluding anything in spiders.txt for even doing an api call, so it never shows in logs & doesn't use up any api hits?

I run a relatively busy & established site with over 8000 products which has been running on zen cart since 2005, so I have a lot of traffic to test this out. So far today, most normal users have scored a 0 which is as expected, but the block log has blocked a fair few really malicious IPs today, so this plugin could prove invaluable for protecting sites from the worst offenders.

johnjlarge:

Yes, I'm on version 2.0 and fully up to date as of just a moment ago with the changes on github.

A couple of ideas, in the abuseipdb_api_call_2023_05.log the IPs of spiders which appear in spiders.txt are still showing as blocked, for example

2023-05-26 16:51:52 IP address 54.236.1.11 API call. Score: 63
(this is pinterest bot, which is still allowed to browse the site, but perhaps we could avoid adding and spider sessions to the api log)

Also, maybe beyond the scope of this plugin, but perhaps we could stop blocked IPs from showing in whos online?

Blocking is working really well, just thinking how the plugin could evolve. Perhaps even excluding anything in spiders.txt for even doing an api call, so it never shows in logs & doesn't use up any api hits?

I run a relatively busy & established site with over 8000 products which has been running on zen cart since 2005, so I have a lot of traffic to test this out. So far today, most normal users have scored a 0 which is as expected, but the block log has blocked a fair few really malicious IPs today, so this plugin could prove invaluable for protecting sites from the worst offenders.

The latest v2.0.4 release of the AbuseIPDB module is now live on GitHub. This update introduces a new feature that allows you to enable or disable known spiders from bypassing IP checks. Additionally, in the previous v2.0.3 release, I added an IP Cleanup feature that automatically deletes expired IP records. You can enable or disable this functionality and configure the IP record expiration period in the admin settings.

I'm facing an issue with the spider detection code. The purpose of this code is to detect spiders/crawlers, if enabled will bypass the api call and creates a log log if enabled as well. However, I've noticed that the log file is not being created even though I can see spiders accessing my site.

I suspect there might be an issue with the spider detection logic or the file logging process. I would greatly appreciate it if someone with more knowledge and experience could take a look at the code snippet and help me identify the problem.

			// Skip API call for known spiders if enabled
				if (isset($spider_flag) && $spider_flag === true && $spider_allow == 'true') {

					// Check if logging is enabled for allowed spiders
						$log_file_name_spiders = 'abuseipdb_spiders_' . date('Y_m') . '.log';
						$log_file_path_spiders = $log_file_path . $log_file_name_spiders;
						$log_message = date('Y-m-d H:i:s') . ' IP address ' . $ip . ' Spider - Score: ' . $abuseScore . PHP_EOL;

					if ($spider_log_enabled == 'true') {			
						file_put_contents($log_file_path_spiders, $log_message, FILE_APPEND);
					}

				return 0; // Return 0 score for spiders or whatever default value you want
				}

I have already verified that the variables ($spider_flag, $spider_allow, $spider_log_enabled, etc.) are correctly set and the paths for the log files are valid.

Could someone please review the code and provide insights into why the log file is not being created? Any suggestions, improvements, or alternative approaches to spider detection in Zen Cart are also welcome.

Thank you in advance for your assistance!

Being lazy and not downloading the plugin, if $log_file_path is set to DIR_FS_LOGS, then the $log_file_name_spiders should have a leading '/' since DIR_FS_LOGS doesn't end in that character.