Suggestion for spam filtering

In ZC1.5.7x the regex line is only checking for HTTPS and FTPS with in the input fields.

While catching spam input coming through my site and information from other networks, I see often things like mysite DOT com, mysite(DOT)com, https://www.mysite.com, http://www.mysite.com then the none payload spammers we see email addresses in non-email fields mailto:[email protected], [email protected].

I changed it to this after testing on a regex tester/debugger. ```
$reg_exUrl = "/(([A-Za-z]{3,9}:(?://)?)(?:[-;:&=+$,\w]+@)?[A-Za-z0-9.-]+|(?:www.|[-;:&=+$,\w]+@)[A-Za-z0-9.-]+|(?:DOT\W)+)/";

  
Not sure if I'm missing the reason for the shorter test but it wasn't catching all of the spam for me..

Also thinking to the future, I'd like to use a configuration field to list the input field names to test.  Mods such as testimonial manager could add it's fields or I just need to update the mod to use the most common names!  I find it's easy to add to the list, but others may not understand it.  Thoughts?

davewest:

the regex line is only checking for HTTPS and FTPS with in the input fields.
Actually, the pattern is: "(https?|ftps?)", where the question-mark means the prior character is optional. So it's checking http, https, ftp, ftps

Your proposed new regex looks overly complicated. Can you explain it in english?

DrByte:

Your proposed new regex looks overly complicated. Can you explain it in english?

First... thanks for moving this, I wasn't sure where to place and didn't really wont to post in full view.

I'm still learning Regex strings and after looking closer at this I think it could be simplified.. Line was getting created as spam came in.. I was capturing spam while testing using the logging system.

This was created using regex101(DOT)com to help explain it.. wrap in code to keep from sanitizing it.

----- Match Information ------

http://www.mysite.com
Match 1
Full match    0-21    http://www.mysite.com
Group 1.    0-21    http://www.mysite.com
Group 2.    0-7    http://

https://www.mysite.com
Match 1
Full match    0-22    https://www.mysite.com
Group 1.    0-22    https://www.mysite.com
Group 2.    0-8    https://

mysite DOT com
Match 1
Full match    7-11    DOT 
Group 1.    7-11    DOT 

mysite(DOT)com
Match 1
Full match    7-11    DOT)
Group 1.    7-11    DOT)

www.mysite.com
Match 1
Full match    0-14    www.mysite.com
Group 1.    0-14    www.mysite.com

mailto:[email protected]
Match 1
Full match    0-26    mailto:[email protected]
Group 1.    0-26    mailto:[email protected]
Group 2.    0-7    mailto:

[email protected]
Match 1
Full match    0-19    [email protected]
Group 1.    0-19    [email protected]

www.url-with-querystring.com/?url=has-querystring
Match 1
Full match    0-28    www.url-with-querystring.com
Group 1.    0-28    www.url-with-querystring.com

------ Full Explanation ---------
1st  Capturing Group  (([A-Za-z]{3,9}:(?:\/\/)?)(?:[-;:&=\+\$,\w]+@)?[A-Za-z0-9.-]+|(?:www.|[-;:&=\+\$,\w]+@)[A-Za-z0-9.-]+|(?:DOT\W)+)
1st Alternative ([A-Za-z]{3,9}:(?:\/\/)?)(?:[-;:&=\+\$,\w]+@)?[A-Za-z0-9.-]+
2nd Capturing Group ([A-Za-z]{3,9}:(?:\/\/)?)
Match a single character present in the list below [A-Za-z]{3,9}
{3,9} Quantifier — Matches between 3 and 9 times, as many times as possible, giving back as needed (greedy)
A-Z a single character in the range between A (index 65) and Z (index 90) (case sensitive)
a-z a single character in the range between a (index 97) and z (index 122) (case sensitive)
: matches the character : literally (case sensitive)
Non-capturing group (?:\/\/)?
? Quantifier — Matches between zero and one times, as many times as possible, giving back as needed (greedy)
\/ matches the character / literally (case sensitive)
\/ matches the character / literally (case sensitive)
Non-capturing group (?:[-;:&=\+\$,\w]+@)?
? Quantifier — Matches between zero and one times, as many times as possible, giving back as needed (greedy)
Match a single character present in the list below [-;:&=\+\$,\w]+
+ Quantifier — Matches between one and unlimited times, as many times as possible, giving back as needed (greedy)
-;:&= matches a single character in the list -;:&= (case sensitive)
\+ matches the character + literally (case sensitive)
\$ matches the character $ literally (case sensitive)
, matches the character , literally (case sensitive)
\w matches any word character (equal to [a-zA-Z0-9_])
@ matches the character @ literally (case sensitive)
Match a single character present in the list below [A-Za-z0-9.-]+
+ Quantifier — Matches between one and unlimited times, as many times as possible, giving back as needed (greedy)
A-Z a single character in the range between A (index 65) and Z (index 90) (case sensitive)
a-z a single character in the range between a (index 97) and z (index 122) (case sensitive)
0-9 a single character in the range between 0 (index 48) and 9 (index 57) (case sensitive)
.- matches a single character in the list .- (case sensitive)
2nd Alternative (?:www.|[-;:&=\+\$,\w]+@)[A-Za-z0-9.-]+
Non-capturing group (?:www.|[-;:&=\+\$,\w]+@)
1st Alternative www.
www matches the characters www literally (case sensitive)
. matches any character (except for line terminators)
2nd Alternative [-;:&=\+\$,\w]+@
Match a single character present in the list below [-;:&=\+\$,\w]+
+ Quantifier — Matches between one and unlimited times, as many times as possible, giving back as needed (greedy)
-;:&= matches a single character in the list -;:&= (case sensitive)
\+ matches the character + literally (case sensitive)
\$ matches the character $ literally (case sensitive)
, matches the character , literally (case sensitive)
\w matches any word character (equal to [a-zA-Z0-9_])
@ matches the character @ literally (case sensitive)
Match a single character present in the list below [A-Za-z0-9.-]+
+ Quantifier — Matches between one and unlimited times, as many times as possible, giving back as needed (greedy)
A-Z a single character in the range between A (index 65) and Z (index 90) (case sensitive)
a-z a single character in the range between a (index 97) and z (index 122) (case sensitive)
0-9 a single character in the range between 0 (index 48) and 9 (index 57) (case sensitive)
.- matches a single character in the list .- (case sensitive)
3rd Alternative (?:DOT\W)+
Non-capturing group (?:DOT\W)+
+ Quantifier — Matches between one and unlimited times, as many times as possible, giving back as needed (greedy)
DOT matches the characters DOT literally (case sensitive)
\W matches any non-word character (equal to [^a-zA-Z0-9_])

I didn't mean a technical explanation of the regex pattern.

I wanted you to explain what your intentions were. Let me ask differently: what's the raw data you based this from?

I "get" that it may be sensible to trap the presence of emails in non-email fields, in some cases. I'd probably suggest that be a different regex pattern and different test independent of URL.

I'm on the fence about whether the substitution of "." for DOT is all that problematic ... mainly because email clients don't auto-convert those to clickable links/addresses.
In your trappings how are these being used? ie: what kinds of fields are they sticking these DOT patterns into?
Does there appear to be logic behind their field choices, or is this just a morphing away from being "caught" as a domain with traditional heuristics?

then the none payload spammersI'm confused about this. What's a "none payload"? Or if it's a typo and you meant "non-payload", I'm still not sure exactly what you meant by the whole sentence.

DrByte:

I didn't mean a technical explanation of the regex pattern.

I wanted you to explain what your intentions were. Let me ask differently: what's the raw data you based this from?
Within the comment field.
About 60% of the spam I've got had url's in full format http;//www 10% was just the https://www.site.com.. some traced back to scams or ads, others advertise there own site.

Another 30% had only email links in the comments 'You can contact me at [email protected]' or trying to hit the auto system mailto:[email protected]

I even had one submitted with (DOT) to try and fool the system!

I'm confused about this. What's a "none payload"? Or if it's a typo and you meant "non-payload", I'm still not sure exactly what you meant by the whole sentence.

Sorry, thought I trashed that line... should of been non-payload..

Noting that a *nice *site to test your regex patterns is https://rubular.com.

@davewest

1. For the DOT handling, I suggest leaving it out of the regex and just str_ireplace(['(DOT)', ' DOT ', '-DOT-', 'DOT'], '.', $var) to convert it to a dot so the rest of the patterns operate normally.

2. I think inspecting for emails should be handled as a separate function, with its own pattern and its own list of fields to inspect. There may be some duplication there, but it allows more specific control. The check for mailto: could probably be left in the URL testing function, with the proviso that it doesn't use the :// syntax.

3. In your trapping are you finding anything that would suggest any benefit from throttling? ie: limit how many messages to submit per minute?

4. How often are you finding things getting through where the only missing factor was URLs without a protocol specified? (with or without www. etc)

1. For the DOT handling, I suggest leaving it out of the regex and just str_ireplace(['(DOT)', ' DOT ', '-DOT-', 'DOT'], '.', $var) to convert it to a dot so the rest of the patterns operate normally.
   I read today where the @ was replaced with a [AT].. With [AT], [DOT] being a social stigma (don't wont the bots to find this) it's really not that big of a deal.. the person that did that was added to my block list and no longer is a problem. I do the DOT so browsers and even this platform from turning intended text links into hyperlinks. Understanding bots are not dumb and can read DOT and AT for what they are..

2. I think inspecting for emails should be handled as a separate function, with its own pattern and its own list of fields to inspect. There may be some duplication there, but it allows more specific control. The check for mailto: could probably be left in the URL testing function, with the proviso that it doesn't use the :// syntax.
   Unfortunate, this is a tough one. I can see a need to allow email formats in this field if there is something keeping this user form contacting through the address in the header or referring to another user. It's also used in signatures of many company's.

3. In your trapping are you finding anything that would suggest any benefit from throttling? ie: limit how many messages to submit per minute? Not sure how that could be done... Bots are creating new session with each submission. However, I use one process when testing that one could fire off as many submissions as it takes to change one character in a email address. Possibly throttling at the observer level could limit the process of mass user accounts.

4. How often are you finding things getting through where the only missing factor was URLs without a protocol specified? (with or without www. etc) Seance ZC154 I've been using user feedback on forms and email blocking. Then with non-CAPTCHA I use my own regex line. So none have made it through. The ones I logged was from contact us and link manager. Most are full http://www some without the http://