Crawler errore 403.shtml existing pages

[diamond1]

Goodmorning everyone,
I have a strange problem that I can not solve and where my provider insists to say that it is not the fault of the server where the site is hosted, but it is certainly the CMS's fault.

I went from http to https and everything works, but from there I believe that when I see the access logs almost all the crawlers (google, bing, facebook) wrapped on existing and non existing pages, the server receives a 403.

66.249.64.155 /403.shtml 06/05/18, 15:34 36199 http://www.MYDOMAIN.com/index.php?ma...create_account
66.249.64.155 /403.shtml 06/05/18, 14:27 36199 http://www.MYDOMAIN.com/bla-bla-bla

Needless to say, the visits have gone down and we are worried,
any help would be useful.

[diamond1]

66.249.64.155 /403.shtml 06/05/18, 15:34 36199 http://www.MYDOMAIN.com/index.php?ma...create_account
66.249.64.155 /403.shtml 06/05/18, 14:27 36199 http://www.MYDOMAIN.com/bla-bla-bla

Normally it would have to do a 301 from http to https instead of a 403 error

MY HTACCES

#Options +FollowSymLinks
Options +SymLinksIfOwnerMatch
RewriteEngine On
RewriteBase /

# da http a https
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} [R=301,L]

# Force WWW & SSL aggiungere wwww a https
RewriteCond %{HTTP_HOST} !^$
RewriteCond %{HTTP_HOST} !^www\. [NC]
RewriteCond %{HTTPS}s ^on(s)|
RewriteRule ^ http%1://www.%{HTTP_HOST}%{REQUEST_URI} [R=301,L]

## BEGIN CEON URI MAPPING REWRITE RULE

# Don't rewrite any URIs ending with a file extension (ending with .[xxxxx])
RewriteCond %{REQUEST_URI} !\.[a-z]{2,5}$ [NC]
# Don't rewrite any URIs for some, popular specific file format extensions,
# which are not covered by main file extension condition above
RewriteCond %{REQUEST_URI} !\.(mp3|mp4|h264)$ [NC]
# Don't rewrite any URIs for some specific file format extensions,
# which are not covered by main file extension condition above
# Uncomment the following line to apply this condition! (Remove the # at the start of the next line)
#RewriteCond %{REQUEST_URI} !\.(3gp|3g2|h261|h263|mj2|mjp2|mp4v|mpg4|m1v|m2v|m4u|f4v|m4v|3dml)$ [NC]
# Don't rewrite admin directory
RewriteCond %{REQUEST_URI} !^/ADMIN[NC]
# Don't rewrite editors directory
RewriteCond %{REQUEST_URI} !^/editors/ [NC]
# Don't rewrite .well-known directory
RewriteCond %{REQUEST_URI} !^/\.well\-known/ [NC]
# Don't rewrite logs directory
RewriteCond %{REQUEST_URI} !^/logs/ [NC]
# Don't rewrite cgi-bin directory
RewriteCond %{REQUEST_URI} !^/cgi\-bin/ [NC]
# Handle all other URIs using Zen Cart (its index.php)
RewriteRule .* index.php [QSA,L]

## END CEON URI MAPPING REWRITE RULE

## rederict link morti

## FINE rederict link morti

RedirectMatch gone "/trigger/*.php$"
RedirectMatch gone "/m/*.php$"
#RewriteRule ^(.*).html$ - [G]

ErrorDocument 404 /index.php?main_page=page_not_found
ErrorDocument 410 /index.php?main_page=page_410
ErrorDocument 403 /index.php?main_page=page_403

# rederict in admin link sito catalogo

# Index Redirect for /index.php?main_page=index and /?main_page=index URL
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /index\.php(\?(main_page=index)?)?\ HTTP/
RewriteRule ^index\.php$ https://www.MYDOMAIN.COM/? [R=301,L]


# compress text, HTML, JavaScript, CSS, and XML
AddOutputFilterByType DEFLATE text/plain
AddOutputFilterByType DEFLATE text/html
AddOutputFilterByType DEFLATE text/xml
AddOutputFilterByType DEFLATE text/css
AddOutputFilterByType DEFLATE application/xml
AddOutputFilterByType DEFLATE application/xhtml+xml
AddOutputFilterByType DEFLATE application/rss+xml
AddOutputFilterByType DEFLATE application/javascript
AddOutputFilterByType DEFLATE application/x-javascript
AddOutputFilterByType DEFLATE application/xml x-font/otf x-font/ttf x-font/eot
AddOutputFilterByType DEFLATE application/xml x-font/woff
AddOutputFilterByType DEFLATE application/xml x-font/woff2

# remove browser bugs
BrowserMatch ^Mozilla/4 gzip-only-text/html
BrowserMatch ^Mozilla/4\.0[678] no-gzip
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
Header append Vary User-Agent

## EXPIRES CACHING ##

ExpiresActive On
ExpiresByType application/vnd.ms-fontobject "access plus 1 year"
ExpiresByType application/x-font-ttf "access plus 1 year"
ExpiresByType application/x-font-opentype "access plus 1 year"
ExpiresByType application/x-font-woff "access plus 1 year"
ExpiresByType application/font-woff "access 1 year"
ExpiresByType image/svg+xml "access plus 1 year"
ExpiresByType image/jpg "access plus 10 days"
ExpiresByType image/jpeg "access plus 10 days"
ExpiresByType text/html "access plus 10 days"
ExpiresByType image/gif "access plus 10 days"
ExpiresByType image/png "access plus 10 days"
ExpiresByType text/css "access plus 10 days"
ExpiresByType application/pdf "access plus 10 days"
ExpiresByType text/x-javascript "access 1 month"
ExpiresByType application/x-shockwave-flash "access plus 10 days"
ExpiresByType image/x-icon "access plus 1 year"
ExpiresDefault "access plus 10 days"

## EXPIRES CACHING ##

<ifModule mod_headers.c>
<filesMatch "\\.(ico|pdf|flv|jpg|jpeg|png|gif|swf)$">
Header set Cache-Control "max-age=864000, public"
</filesMatch>

<filesMatch "\\.(css)$">
Header set Cache-Control "max-age=864000, public"
</filesMatch>

<filesMatch "\\.(js)$">
Header set Cache-Control "max-age=864000, private"
</filesMatch>

<filesMatch "\\.(xml|txt)$">
Header set Cache-Control "max-age=864000, public, must-revalidate"
</filesMatch>

<filesMatch "\\.(html|htm|php)$">
Header set Cache-Control "max-age=1, private, must-revalidate"
</filesMatch>
</ifModule>

<Files php.ini>
order allow,deny
deny from all
</Files>

# php -- BEGIN cPanel-generated handler, do not edit
<IfModule fcgid_module>
<IfModule mime_module>
AddHandler fcgid-script .php .php7 .phtml
</IfModule>
</IfModule>

# php -- END cPanel-generated handler, do not edit

Needless to say, the visits have gone down and we are worried,
any help would be useful.

[diamond1]

Perhaps these errors can help you understand where the problem is

what does my say have not compatible SSL

Tue Apr 17 22:19:17.227126 2018] [access_compat:error] [pid 24449:tid 140656194987776] [client 66.249.64.81:56044] AH01797: client denied by server configuration: /home/paparaz/public_html/includes/templates/mobishop_blue, referer: http://www.MYDOMAIN:COM/index.php?main_page=product_info&cPath=255&products_id=552
[Wed Apr 18 00:19:03.706294 2018] [ssl:error] [pid 27350:tid 140656076908288] AH02032: Hostname www.whitleygalleries.com provided via SNI and hostname www.MYDOMAIN:COM provided via HTTP have no compatible SSL setup

[diamond1]

Excuse me if I insist nobody knows what could cause this 403?

[gilby]

maybe a conflict with your commercial template "mobishop_blue" ?

Check that your SSL is installed ok
SSL Check

[diamond1]

Thank you for answering me, no because the templates in question are disabled for years even there are not even the files on the server. We are on a shared server also if you see this error which is a page that exists only that has been requested in http instead of https gives me error 403 instead of 301
66.249.64.155 /403.shtml 06/05/18, 15:34 36199 http://www.MYDOMAIN.com/index.php?ma...create_account

On the SSL test he tells me an A and gives me this that I do not understand if it's right.

Server Key and Certificate #1
Subject www.MYDOMAIN.COM
Fingerprint SHA256: bf1eb77d4717b424930d5f5bb941f81557b22f1bf4eae90f09d9c1be68e41ae8
Pin SHA256: QHIWDqUlQfRSpPxktXjhGQjSR6sI/Bn++UoSdEe3OUc=
Common names www.MYDOMAIN.COM
Alternative names www.MYDOMAIN.COM MYDOMAIN.COM
Serial Number 0294c0c8550ba6f09c7676dd88575f48
Valid from Fri, 13 Apr 2018 00:00:00 UTC
Valid until Sat, 13 Apr 2019 12:00:00 UTC (expires in 11 months and 4 days)
Key RSA 2048 bits (e 65537)
Weak key (Debian) No
Issuer Thawte RSA CA 2018
AIA: http://cacerts.thawte.com/ThawteRSACA2018.crt
Signature algorithm SHA256withRSA
Extended Validation No
Certificate Transparency Yes (certificate)
OCSP Must Staple No
Revocation information CRL, OCSP
CRL: http://cdp.thawte.com/ThawteRSACA2018.crl
OCSP: http://status.thawte.com
Revocation status Good (not revoked)
DNS CAA No (more info)
Trusted Yes
Mozilla Apple Android Java Windows


Additional Certificates (if supplied)
Certificates provided 2 (2684 bytes)
Chain issues None
#2
Subject Thawte RSA CA 2018
Fingerprint SHA256: 9a5eecee9c7d898bd81dc3bf066daf6aefb8db1c59676206d2bfdd682312c6f6
Pin SHA256: S0mHTmqv2QhJEfy5vyPVERSnyMEliJzdC8RXduOjhAs=
Valid until Sat, 06 Nov 2027 12:23:52 UTC (expires in 9 years and 5 months)
Key RSA 2048 bits (e 65537)
Issuer DigiCert Global Root CA
Signature algorithm SHA256withRSA


Show Certification Paths Certification Paths
Click here to expand

Certificate #2: RSA 2048 bits (SHA256withRSA) No SNI Show Server Certificate
Click here to expand

Configuration

Protocols
TLS 1.3 No
TLS 1.2 Yes
TLS 1.1 No
TLS 1.0 No
SSL 3 No
SSL 2 No
For TLS 1.3 tests, we currently support draft version 18.



Cipher Suites
# TLS 1.2 (server has no preference)
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (0xc027) ECDH sect571r1 (eq. 15360 bits RSA) FS 128
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f) ECDH sect571r1 (eq. 15360 bits RSA) FS 128
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (0xc028) ECDH sect571r1 (eq. 15360 bits RSA) FS 256
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (0xc030) ECDH sect571r1 (eq. 15360 bits RSA) FS 256


Handshake Simulation
Android 4.4.2 RSA 2048 (SHA256) TLS 1.2 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 ECDH secp521r1 FS
Android 5.0.0 RSA 2048 (SHA256) TLS 1.2 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 ECDH secp521r1 FS
Android 6.0 RSA 2048 (SHA256) TLS 1.2 > http/1.1 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 ECDH secp256r1 FS
Android 7.0 RSA 2048 (SHA256) TLS 1.2 > h2 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 ECDH secp256r1 FS
BingPreview Jan 2015 RSA 2048 (SHA256) TLS 1.2 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 ECDH sect571r1 FS
Chrome 49 / XP SP3 RSA 2048 (SHA256) TLS 1.2 > h2 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 ECDH secp256r1 FS
Chrome 57 / Win 7 R RSA 2048 (SHA256) TLS 1.2 > h2 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 ECDH secp256r1 FS
Firefox 31.3.0 ESR / Win 7 RSA 2048 (SHA256) TLS 1.2 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 ECDH secp256r1 FS
Firefox 47 / Win 7 R RSA 2048 (SHA256) TLS 1.2 > h2 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 ECDH secp256r1 FS
Firefox 49 / XP SP3 RSA 2048 (SHA256) TLS 1.2 > h2 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 ECDH secp256r1 FS
Firefox 53 / Win 7 R RSA 2048 (SHA256) TLS 1.2 > h2 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 ECDH secp256r1 FS
Googlebot Feb 2018 RSA 2048 (SHA256) TLS 1.2 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 ECDH secp256r1 FS
IE 11 / Win 7 R RSA 2048 (SHA256) TLS 1.2 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 ECDH secp256r1 FS
IE 11 / Win 8.1 R RSA 2048 (SHA256) TLS 1.2 > http/1.1 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 ECDH secp256r1 FS
IE 11 / Win Phone 8.1 R RSA 2048 (SHA256) TLS 1.2 > http/1.1 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 ECDH secp256r1 FS
IE 11 / Win Phone 8.1 Update R RSA 2048 (SHA256) TLS 1.2 > http/1.1 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 ECDH secp256r1 FS
IE 11 / Win 10 R RSA 2048 (SHA256) TLS 1.2 > h2 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 ECDH secp256r1 FS
Edge 15 / Win 10 R RSA 2048 (SHA256) TLS 1.2 > h2 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 ECDH secp256r1 FS
Edge 13 / Win Phone 10 R RSA 2048 (SHA256) TLS 1.2 > h2 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 ECDH secp256r1 FS
Java 8u161 RSA 2048 (SHA256) TLS 1.2 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 ECDH secp256r1 FS
OpenSSL 1.0.1l R RSA 2048 (SHA256) TLS 1.2 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 ECDH sect571r1 FS
OpenSSL 1.0.2e R RSA 2048 (SHA256) TLS 1.2 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 ECDH secp256r1 FS
Safari 6 / iOS 6.0.1 RSA 2048 (SHA256) TLS 1.2 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 ECDH secp256r1 FS
Safari 7 / iOS 7.1 R RSA 2048 (SHA256) TLS 1.2 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 ECDH secp256r1 FS
Safari 7 / OS X 10.9 R RSA 2048 (SHA256) TLS 1.2 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 ECDH secp256r1 FS
Safari 8 / iOS 8.4 R RSA 2048 (SHA256) TLS 1.2 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 ECDH secp256r1 FS
Safari 8 / OS X 10.10 R RSA 2048 (SHA256) TLS 1.2 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 ECDH secp256r1 FS
Safari 9 / iOS 9 R RSA 2048 (SHA256) TLS 1.2 > h2 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 ECDH secp256r1 FS
Safari 9 / OS X 10.11 R RSA 2048 (SHA256) TLS 1.2 > h2 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 ECDH secp256r1 FS
Safari 10 / iOS 10 R RSA 2048 (SHA256) TLS 1.2 > h2 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 ECDH secp256r1 FS
Safari 10 / OS X 10.12 R RSA 2048 (SHA256) TLS 1.2 > h2 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 ECDH secp256r1 FS
Apple ATS 9 / iOS 9 R RSA 2048 (SHA256) TLS 1.2 > h2 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 ECDH secp256r1 FS
Yahoo Slurp Jan 2015 RSA 2048 (SHA256) TLS 1.2 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 ECDH secp384r1 FS
YandexBot Jan 2015 RSA 2048 (SHA256) TLS 1.2 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 ECDH sect571r1 FS
# Not simulated clients (Protocol mismatch)
Click here to expand

(1) Clients that do not support Forward Secrecy (FS) are excluded when determining support for it.
(2) No support for virtual SSL hosting (SNI). Connects to the default site if the server uses SNI.
(3) Only first connection attempt simulated. Browsers sometimes retry with a lower protocol version.
(R) Denotes a reference browser or client, with which we expect better effective security.
(All) We use defaults, but some platforms do not use their best protocols and features (e.g., Java 6 & 7, older IE).
(All) Certificate trust is not checked in handshake simulation, we only perform TLS handshake.



Protocol Details
DROWN No, server keys and hostname not seen elsewhere with SSLv2
(1) For a better understanding of this test, please read this longer explanation
(2) Key usage data kindly provided by the Censys network search engine; original DROWN website here
(3) Censys data is only indicative of possible key and certificate reuse; possibly out-of-date and not complete
Secure Renegotiation Supported
Secure Client-Initiated Renegotiation No
Insecure Client-Initiated Renegotiation No
BEAST attack Mitigated server-side (more info)
POODLE (SSLv3) No, SSL 3 not supported (more info)
POODLE (TLS) No (more info)
Downgrade attack prevention Unknown (requires support for at least two protocols, excl. SSL2)
SSL/TLS compression No
RC4 No
Heartbeat (extension) Yes
Heartbleed (vulnerability) No (more info)
Ticketbleed (vulnerability) No (more info)
OpenSSL CCS vuln. (CVE-2014-0224) No (more info)
OpenSSL Padding Oracle vuln.
(CVE-2016-2107) No (more info)
ROBOT (vulnerability) No (more info)
Forward Secrecy Yes (with most browsers) ROBUST (more info)
ALPN Yes h2 http/1.1
NPN No
Session resumption (caching) Yes
Session resumption (tickets) Yes
OCSP stapling Yes
Strict Transport Security (HSTS) No
HSTS Preloading Not in: Chrome Edge Firefox IE
Public Key Pinning (HPKP) No (more info)
Public Key Pinning Report-Only No
Public Key Pinning (Static) No (more info)
Long handshake intolerance No
TLS extension intolerance No
TLS version intolerance No
Incorrect SNI alerts No
Uses common DH primes No, DHE suites not supported
DH public server param (Ys) reuse No, DHE suites not supported
ECDH public server param reuse No
Supported Named Groups sect283k1, sect283r1, sect409k1, sect409r1, sect571k1, sect571r1, secp256k1, secp256r1, secp384r1, secp521r1, brainpoolP256r1, brainpoolP384r1, brainpoolP512r1 (Server has no preference)
SSL 2 handshake compatibility No


HTTP Requests
1 https://www.MYDOMAIN.COM/ (HTTP/1.1 200 OK)



Miscellaneous
Test date Tue, 08 May 2018 13:06:07 UTC
Test duration 62.713 seconds
HTTP status code 200
HTTP server signature Apache
Server hostname imalia.dnshigh.com


One thing is safe since we installed the certificate visits that come from google or other search engines if they do not have https as refere show a 403 error but only for them customers do not see the 403 go directly to the page.

With the ssl verification I have these as errors

Android 2.3.7 No SNI 2 Protocol mismatch (not simulated)
Android 4.0.4 Protocol mismatch (not simulated)
Android 4.1.1 Protocol mismatch (not simulated)
Android 4.2.2 Protocol mismatch (not simulated)
Android 4.3 Protocol mismatch (not simulated)
########## Jan 2015 Protocol mismatch (not simulated)
IE 6 / XP No FS 1 No SNI 2 Protocol mismatch (not simulated)
IE 7 / Vista Protocol mismatch (not simulated)
IE 8 / XP No FS 1 No SNI 2 Protocol mismatch (not simulated)
IE 8-10 / Win 7 R Protocol mismatch (not simulated)
IE 10 / Win Phone 8.0 Protocol mismatch (not simulated)
Java 6u45 No SNI 2 Protocol mismatch (not simulated)
Java 7u25 Protocol mismatch (not simulated)
OpenSSL 0.9.8y Protocol mismatch (not simulated)
Safari 5.1.9 / OS X 10.6.8 Protocol mismatch (not simulated)
Safari 6.0.4 / OS X 10.8.4 R Protocol mismatch (not simulated)
(1) Clients that do not support Forward Secrecy (FS) are excluded when determining support for it.
(2) No support for virtual SSL hosting (SNI). Connects to the default site if the server uses SNI.
(3) Only first connection attempt simulated. Browsers sometimes retry with a lower protocol version.
(R) Denotes a reference browser or client, with which we expect better effective security.
(All) We use defaults, but some platforms do not use their best protocols and features (e.g., Java 6 & 7, older IE).
(All) Certificate trust is not checked in handshake simulation, we only perform TLS handshake.

[dbltoe]

See if this doesn't help. https://forums.cpanel.net/threads/ac...-error.449032/

[diamond1]

See if this doesn't help. https://forums.cpanel.net/threads/ac...-error.449032/

It did not help me, we do not know what to do the visits have gone down 80% now it's been more than a month that we installed the certificate, and I'm thinking of changing hosting since they do not have a solution they think it's the CSM or some htacces. But why does 403 only happen you have boot?

[diamond1]

RESOLVED,
saw the severity and not having been able to solve the problem with serverplan we decided to move everything on another web hosting.

[gilby]

RESOLVED,
saw the severity and not having been able to solve the problem with serverplan we decided to move everything on another web hosting.

Often the best way to solve a "clueless" host problem