Crawl Errors

**BoydBreen** · 13 Feb 2014, 09:42 PM

Hi Everyone

I keep getting allot of crawl errors for dynamic pages all the pages listed below are action pages or info pages that should not be listed or being crawled. I get the following response from google.

The target URL doesn't exist, but your server is not returning a 404 (file not found) error. Learn more
Your server is redirecting requests for a non-existent page, instead of returning a 404 response code. This creates a poor experience for searchers and search engines.

Can anyone point me in the right direction here perhaps there is something ajar with my ht access file I am not sure any help would be greatly appreciated. I am using ultimate SEO URL plugin but this is not a historic problem so I have ruled this out.

Sample of pages

index.php?main_page=login
2/10/14

2
swivel-and-tilt-lcd-plasma-tv-bracket-42-inch-to-60-inch-p-66.html?action=buy_now
2/7/14

3
white-mk-51-wall-speaker-plate-architrave-set-p-69.html?action=buy_now&sort=20a
2/8/14

4
white-mk-architrave-single-speaker-wall-plate-p-46.html?action=buy_now
2/4/14

5
future-automation-lslefa-lsmefa-tv-lift-with-flap-actuator-p-133.html?action=buy_now
2/7/14

6
white-single-gang-double-hdmi-wall-plate-p-32.html?action=buy_now
2/5/14

7
low-level-cabinet-chest-tv-lift-la-p-77.html?action=notify
2/8/14

8
ctouch-65-led-interactive-flat-screen-touch-panel-p-94.html?action=buy_now&id=9
2/5/14

9
casio-xja251uj-wxga-led-projector-p-129.html?action=buy_now
2/6/14

10
white-mk-architrave-single-speaker-wall-plate-p-46.html?action=notify&language=en
2/7/14

11
custom-audio-and-multi-room-audio-servcie-p-7.html?action=buy_now&cpath=1
2/8/14

12
hitachi-cpaw3019wnm-wxga-ultra-short-throw-projector-p-119.html?action=notify
2/6/14

13
venset-universal-vesa-mounting-plate-p-121.html?action=buy_now
2/4/14

14
casio-xja141uj-led-projector-p-124.html?action=notify
2/5/14

15
hidden-cabling-installation-service-p-2.html?action=notify
2/8/14

16
casio-xja141uj-xga-led-projector-p-124.html?action=buy_now&language=en
2/5/14

17
casio-xja246uj-wireless-wxga-led-projector-p-128.html?action=buy_now&language=en
2/5/14

18
casio-xja150vej-led-projector-p-126.html?action=buy_now&language=en
2/5/14

19
casio-xja131uj-led-projector-p-123.html?action=buy_now&language=en
2/5/14

20
hitachi-cpaw252wnm-compact-short-throw-projector-p-118.html?action=buy_now&manufacturers_id=2
2/7/14

21
venset-universal-vesa-mounting-plate-p-121.html?action=buy_now&manufacturers_id=1
2/6/14

22
white-single-gang-plastic-brush-face-plate-with-white-brushes-p-20.html?action=buy_now&language=en&disp_order=1&page=5
2/8/14

23
satin-steel-single-hdmi-face-plate-p-36.html?action=notify&language=en
2/8/14

24
casio-xja246uj-wireless-wxga-led-projector-p-128.html?action=buy_now&sort=3d&language=en
2/7/14

25
hitachi-cpa302nm-3000-lum-short-throw-projector-p-115.html?action=buy_now&language=en&id=16
2/6/14

**kobra** · 15 Feb 2014, 03:22 PM

I am using ultimate SEO URL plugin but this is not a historic problem so I have ruled this out.

Try turning it off and see what happens

**mc12345678** · 15 Feb 2014, 06:10 PM

Originally Posted by BoydBreen

Hi Everyone

The target URL doesn't exist, but your server is not returning a 404 (file not found) error. Learn more
Your server is redirecting requests for a non-existent page, instead of returning a 404 response code. This creates a poor experience for searchers and search engines.

Haven't personally tried to access the problem pages, but depending on how your system responds, v1.5.1 and below have an issue with the product not found page that might clear up some of what is described.

Originally Posted by BoydBreen

perhaps there is something ajar with my ht access file I am not sure any help would be greatly appreciated. I am using ultimate SEO URL plugin but this is not a historic problem so I have ruled this out.

I hate to say it, but the first does not go with the last.... Meaning, there is probably something that has changed either in the htaccess or in a file related to ultimate SEO. What has changed in your system that may have caused this? PHP upgrade? template change? added/modified anything about your .htaccess? Once the plugin is off and your site is verified to work, suggest verifying that the URI rewriter is up-to-date and installed correctly.

**BoydBreen** · 19 Feb 2014, 07:27 PM

Hi Mc

thanks for your response.

I have copied my .htaccess file below I have highlighted an area in red this was the most recent addition to this file ##### NOTE: Replace /shop/ with the relative web path of your catalog in the "Rewrite Base" line below:

Options +FollowSymLinks
RewriteEngine On
RewriteBase /

### start of added code ###

# Rewrite to www
RewriteCond %{HTTP_HOST} ^avinstalls.co.uk [NC]
RewriteRule ^(.*)$ http://www.avinstalls.co.uk/$1 [R=301,L]

# Redirect index.html and index.htm to /
RewriteRule ^default\.htm?$ / [NC,R,L]
RewriteRule ^index\.htm?$ / [NC,R,L]

### end of added code ###

### START ###
### rewrite all links from / to .html ###
RewriteRule (.+)/$ /$1.html [L,R=301]
### END ###

# From Ultimate SEO URLs
RewriteRule ^(.*)-c-([^/])/(.*)-p-(.*).html$ index\.php?main_page=product_info&products_id=$4&cpath=$2 [L]
RewriteRule ^(.*)-p-(.*).html$ index\.php?main_page=product_info&products_id=$2&%{QUERY_STRING} [L]

### start of edited code ###

# ENABLED: Normal Ultimate SEO URLs (disable if using ADD_PRODUCT_CAT)
RewriteRule ^(.*)-c-(.*).html$ index\.php?main_page=index&cPath=$2&%{QUERY_STRING} [L]

# DISABLED (notice the # before the following 3 RewriteRules. Note: ADD_PRODUCT_CAT Ultimate SEO URLs (disable if not using ADD_PRODUCT_CAT)
# RewriteRule ^(.*)-c-(.*).html$ $1-c-$2/ [R=301,L]
# RewriteRule ^(.*)-c-([^/]+)$ %{REQUEST_URI}/ [R=301,L]
# RewriteRule ^(.*)-c-([^/]+)/$ index\.php?main_page=index&cPath=$2&%{QUERY_STRING} [L]

### end of edited code ###

RewriteRule ^(.*)-m-([0-9]+).html$ index\.php?main_page=index&manufacturers_id=$2&%{QUERY_STRING} [L]
RewriteRule ^(.*)-pi-([0-9]+).html$ index\.php?main_page=popup_image&pID=$2&%{QUERY_STRING} [L]
RewriteRule ^(.*)-pr-([0-9]+).html$ index\.php?main_page=product_reviews&products_id=$2&%{QUERY_STRING} [L]
RewriteRule ^(.*)-pri-([0-9]+).html$ index\.php?main_page=product_reviews_info&products_id=$2&%{QUERY_STRING} [L]
RewriteRule ^(.*)-ezp-([0-9]+).html$ index\.php?main_page=page&id=$2&%{QUERY_STRING} [L]

# All other pages
# Don't rewrite real files or directories
RewriteCond %{REQUEST_FILENAME} !-f [NC]
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*).html$ index\.php?main_page=$1&%{QUERY_STRING} [L]

# Handle when a resource is not found (404 Error Page)
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ index\.php?main_page=page_not_found&%{QUERY_STRING} [L]

Redirect 301 /testimonials.htm http://www.avinstalls.co.uk/index.ph...page=page&id=3
Redirect 301 /sector_homecinema.htm http://www.avinstalls.co.uk/index.ph...=index&cPath=1
Redirect 301 /sector_retailleisure.htm http://www.avinstalls.co.uk/index.ph...ndex&cPath=5_9
Redirect 301 /sector_education.htm http://www.avinstalls.co.uk/index.ph...ndex&cPath=5_6
Redirect 301 /sector_corporate.htm http://www.avinstalls.co.uk/index.ph...ndex&cPath=5_7
Redirect 301 /sector_church.htm http://www.avinstalls.co.uk/index.ph...ndex&cPath=5_8
Redirect 301 /gallery.htm http://www.avinstalls.co.uk/index.ph...page=page&id=4
Redirect 301 /contact.htm http://www.avinstalls.co.uk/index.ph...age=contact_us

All of this was a more recent addition

# compress text, HTML, JavaScript, CSS, and XML
AddOutputFilterByType DEFLATE text/plain
AddOutputFilterByType DEFLATE text/html
AddOutputFilterByType DEFLATE text/xml
AddOutputFilterByType DEFLATE text/css
AddOutputFilterByType DEFLATE application/xml
AddOutputFilterByType DEFLATE application/xhtml+xml
AddOutputFilterByType DEFLATE application/rss+xml
AddOutputFilterByType DEFLATE application/javascript
AddOutputFilterByType DEFLATE application/x-javascript

# remove browser bugs
BrowserMatch ^Mozilla/4 gzip-only-text/html
BrowserMatch ^Mozilla/4\.0[678] no-gzip
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
Header append Vary User-Agent

## EXPIRES CACHING ##

ExpiresActive On
ExpiresByType image/jpg "access plus 7 days"
ExpiresByType image/jpeg "access plus 7 days"
ExpiresByType image/gif "access plus 1 month"
ExpiresByType image/png "access plus 7 days"
ExpiresByType text/css "access plus 1 month"
ExpiresByType application/pdf "access plus 1 month"
ExpiresByType text/x-javascript "access plus 1 month"
ExpiresByType application/x-shockwave-flash "access plus 1 month"
ExpiresByType image/x-icon "access plus 7 days"
ExpiresDefault "access plus 2 days"

## EXPIRES CACHING ##

<IfModule mod_expires.c>
# Enable expirations
ExpiresActive On
# Default directive
ExpiresDefault "access plus 1 month"
# My favicon
ExpiresByType image/x-icon "access plus 1 year"
# Images
ExpiresByType image/gif "access plus 1 month"
ExpiresByType image/png "access plus 1 month"
ExpiresByType image/jpg "access plus 1 month"
ExpiresByType image/jpeg "access plus 1 month"
# CSS
ExpiresByType text/css "access 1 month"
# Javascript
ExpiresByType application/javascript "access plus 1 month"
</IfModule>

I am not the best with .htaccess we have had problems in the past but were running fine for a long time more than 12 months I did add some caching codes ect that i got from gmetrics to help speed things up perhaps this is causing the problem its they are the only bits that have been recently added.

Any help is greatly appreciated.

Boyd

**mc12345678** · 19 Feb 2014, 08:11 PM

Originally Posted by BoydBreen

Hi Mc

thanks for your response.

I have copied my .htaccess file below I have highlighted an area in red this was the most recent addition to this file ##### NOTE: Replace /shop/ with the relative web path of your catalog in the "Rewrite Base" line below:

Options +FollowSymLinks
RewriteEngine On
RewriteBase /

### start of added code ###

# Rewrite to www
RewriteCond %{HTTP_HOST} ^avinstalls.co.uk [NC]
RewriteRule ^(.*)$ http://www.avinstalls.co.uk/$1 [R=301,L]

# Redirect index.html and index.htm to /
RewriteRule ^default\.htm?$ / [NC,R,L]
RewriteRule ^index\.htm?$ / [NC,R,L]

### end of added code ###

### START ###
### rewrite all links from / to .html ###
RewriteRule (.+)/$ /$1.html [L,R=301]
### END ###

# From Ultimate SEO URLs
RewriteRule ^(.*)-c-([^/])/(.*)-p-(.*).html$ index\.php?main_page=product_info&products_id=$4&cpath=$2 [L]
RewriteRule ^(.*)-p-(.*).html$ index\.php?main_page=product_info&products_id=$2&%{QUERY_STRING} [L]

### start of edited code ###

# ENABLED: Normal Ultimate SEO URLs (disable if using ADD_PRODUCT_CAT)
RewriteRule ^(.*)-c-(.*).html$ index\.php?main_page=index&cPath=$2&%{QUERY_STRING} [L]

# DISABLED (notice the # before the following 3 RewriteRules. Note: ADD_PRODUCT_CAT Ultimate SEO URLs (disable if not using ADD_PRODUCT_CAT)
# RewriteRule ^(.*)-c-(.*).html$ $1-c-$2/ [R=301,L]
# RewriteRule ^(.*)-c-([^/]+)$ %{REQUEST_URI}/ [R=301,L]
# RewriteRule ^(.*)-c-([^/]+)/$ index\.php?main_page=index&cPath=$2&%{QUERY_STRING} [L]

### end of edited code ###

RewriteRule ^(.*)-m-([0-9]+).html$ index\.php?main_page=index&manufacturers_id=$2&%{QUERY_STRING} [L]
RewriteRule ^(.*)-pi-([0-9]+).html$ index\.php?main_page=popup_image&pID=$2&%{QUERY_STRING} [L]
RewriteRule ^(.*)-pr-([0-9]+).html$ index\.php?main_page=product_reviews&products_id=$2&%{QUERY_STRING} [L]
RewriteRule ^(.*)-pri-([0-9]+).html$ index\.php?main_page=product_reviews_info&products_id=$2&%{QUERY_STRING} [L]
RewriteRule ^(.*)-ezp-([0-9]+).html$ index\.php?main_page=page&id=$2&%{QUERY_STRING} [L]

# All other pages
# Don't rewrite real files or directories
RewriteCond %{REQUEST_FILENAME} !-f [NC]
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*).html$ index\.php?main_page=$1&%{QUERY_STRING} [L]

# Handle when a resource is not found (404 Error Page)
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ index\.php?main_page=page_not_found&%{QUERY_STRING} [L]

Redirect 301 /testimonials.htm http://www.avinstalls.co.uk/index.ph...page=page&id=3
Redirect 301 /sector_homecinema.htm http://www.avinstalls.co.uk/index.ph...=index&cPath=1
Redirect 301 /sector_retailleisure.htm http://www.avinstalls.co.uk/index.ph...ndex&cPath=5_9
Redirect 301 /sector_education.htm http://www.avinstalls.co.uk/index.ph...ndex&cPath=5_6
Redirect 301 /sector_corporate.htm http://www.avinstalls.co.uk/index.ph...ndex&cPath=5_7
Redirect 301 /sector_church.htm http://www.avinstalls.co.uk/index.ph...ndex&cPath=5_8
Redirect 301 /gallery.htm http://www.avinstalls.co.uk/index.ph...page=page&id=4
Redirect 301 /contact.htm http://www.avinstalls.co.uk/index.ph...age=contact_us

All of this was a more recent addition

# compress text, HTML, JavaScript, CSS, and XML
AddOutputFilterByType DEFLATE text/plain
AddOutputFilterByType DEFLATE text/html
AddOutputFilterByType DEFLATE text/xml
AddOutputFilterByType DEFLATE text/css
AddOutputFilterByType DEFLATE application/xml
AddOutputFilterByType DEFLATE application/xhtml+xml
AddOutputFilterByType DEFLATE application/rss+xml
AddOutputFilterByType DEFLATE application/javascript
AddOutputFilterByType DEFLATE application/x-javascript

# remove browser bugs
BrowserMatch ^Mozilla/4 gzip-only-text/html
BrowserMatch ^Mozilla/4\.0[678] no-gzip
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
Header append Vary User-Agent

## EXPIRES CACHING ##

ExpiresActive On
ExpiresByType image/jpg "access plus 7 days"
ExpiresByType image/jpeg "access plus 7 days"
ExpiresByType image/gif "access plus 1 month"
ExpiresByType image/png "access plus 7 days"
ExpiresByType text/css "access plus 1 month"
ExpiresByType application/pdf "access plus 1 month"
ExpiresByType text/x-javascript "access plus 1 month"
ExpiresByType application/x-shockwave-flash "access plus 1 month"
ExpiresByType image/x-icon "access plus 7 days"
ExpiresDefault "access plus 2 days"

## EXPIRES CACHING ##

<IfModule mod_expires.c>
# Enable expirations
ExpiresActive On
# Default directive
ExpiresDefault "access plus 1 month"
# My favicon
ExpiresByType image/x-icon "access plus 1 year"
# Images
ExpiresByType image/gif "access plus 1 month"
ExpiresByType image/png "access plus 1 month"
ExpiresByType image/jpg "access plus 1 month"
ExpiresByType image/jpeg "access plus 1 month"
# CSS
ExpiresByType text/css "access 1 month"
# Javascript
ExpiresByType application/javascript "access plus 1 month"
</IfModule>

I am not the best with .htaccess we have had problems in the past but were running fine for a long time more than 12 months I did add some caching codes ect that i got from gmetrics to help speed things up perhaps this is causing the problem its they are the only bits that have been recently added.

Any help is greatly appreciated.

Boyd

I can't say that I have done a thorough review of theprovided htaccess, partially because I didn't see any red highlighted text/code, but if the following was what was added, then perhaps the sequence/location of it is causing a problem:

Code:

### start of edited code ###

# ENABLED: Normal Ultimate SEO URLs (disable if using ADD_PRODUCT_CAT)
RewriteRule ^(.*)-c-(.*).html$ index\.php?main_page=index&cPath=$2&%{QUERY_STRING} [L]

# DISABLED (notice the # before the following 3 RewriteRules. Note: ADD_PRODUCT_CAT Ultimate SEO URLs (disable if not using ADD_PRODUCT_CAT)
# RewriteRule ^(.*)-c-(.*).html$ $1-c-$2/ [R=301,L]
# RewriteRule ^(.*)-c-([^/]+)$ %{REQUEST_URI}/ [R=301,L]
# RewriteRule ^(.*)-c-([^/]+)/$ index\.php?main_page=index&cPath=$2&%{QUERY_STRING} [L]

### end of edited code ###

There are online htaccess review tools that could possibly help out as well. Basically drop in the htaccess file, then provide a web address to test, and it steps through the htaccess and identifies where a result would be provided that would cause the htaaccess to be reprocessed. So then submit that new result and let it run to the next end point, etc. Until have reached the end of the file or get kicked out of the htaccess.

Partially thinking that the above code should be placed further down in the htaccess file. Maybe someone else may see something more telling.

**lhungil** · 20 Feb 2014, 12:02 AM

That looks like a really old version of the module as well. May be worth the time to upgrade and clean up the .htaccess file.

**mc12345678** · 20 Feb 2014, 12:44 AM

Forgot to mention, is there some reason that using rewrite to ADD .html onto the address? Why not just leave it off? (Seems like that is an option of the rewriter.)

**BoydBreen** · 3 Mar 2014, 02:47 PM

HI Guys

Thanks for all you help,

I have upgraded to the latest ultimate SEO URLs and changed my .htaccess file, I have also reset google webmasters tools, however I am still getting the same crawl errors, is there anywhere else I should be looking for a problem other than with the ultimate seo urls or the ht access file ?

Many Thanks

Boyd

**lhungil** · 3 Mar 2014, 06:38 PM

Depends, if you can browse no problem and the listed URLs are valid and work... I'd not overly worry... If the URLs are not valid and from your site, I'd look into the source more... If not from your site, see if you can update them or if heavily accessed redirect to an appropriate page...

**BoydBreen** · 8 Apr 2014, 09:18 AM

Hi Everyone

Thanks to anyone who has contributed so far.

I am still having the same problem in that I am still getting a massive amount of soft 404 errors all based around the URL an a call to action I have copied a few below.
Can anyone tell me why these pages are continue to return soft 404 errors the links work and I am confused as to why they are even being flagged up none of these pages are indexed by google. When I fetch the page as google they return HTTP/1.1 302 Moved Temporarily which I believe to be correct can anyone shed any light on why this may be happening?

projector-and-manual-screen-with-installation-p-83.html?action=buy_now&language=en
3/25/14

5
hidden-cabling-installation-service-p-2.html?action=buy_now
4/5/14

6
swivel-and-tilt-lcd-plasma-tv-bracket-42-inch-to-60-inch-p-66.html?action=buy_now
3/6/14

7
hidden-cabling-installation-service-p-2.html?action=buy_now&language=en
3/10/14

8
venset-ts1000a-motorised-television-screen-lift-p-14.html?action=buy_now
3/30/14

9
white-double-gang-plastic-brush-face-plate-with-black-brushes-p-21.html?action=buy_now

Thread: Crawl Errors

Thread Tools

Search Thread

Display

Crawl Errors

Re: Crawl Errors

Re: Crawl Errors

Re: Crawl Errors

Re: Crawl Errors

Re: Crawl Errors

Re: Crawl Errors

Re: Crawl Errors

Re: Crawl Errors

Re: Crawl Errors

Posting Permissions