Forums / Search Engines / Indexed, though blocked by robots.txt

Indexed, though blocked by robots.txt

Views: 66

Results 1 to 4 of 4
17 Feb 2021, 2:14 AM
#1
lloyd_borrett avatar

lloyd_borrett

Zen Follower

Join Date:
Mar 2006
Location:
Rosebud, Victoria, Australia
Posts:
308
Plugin Contributions:
2

Indexed, though blocked by robots.txt

Google Search Console was warning me:
Coverage > Indexed, though blocked by robots.txt

Affected pages: 1.57K
The number of warnings has increased significantly recently. Google probably changed a policy.

The warnings are mostly for things like:
https://www.scubadoctor.com.au/diveshop/index.php?main_page=popup_image&pID=4896
https://www.scubadoctor.com.au/diveshop/index.php?main_page=popup_image_additional&pID=4612&pic=2&products_image_large_additional=images/ocean-design/ocean-design-moana-msf-pkg_03.jpg
https://www.scubadoctor.com.au/diveshop/index.php?main_page=product_reviews_write&products_id=5691

I can see why it might be useful to let it index the additional images. So I changed my robots.txt file.

Instead of just…
Disallow: /diveshop/index.php?main_page=popup_image*

I now have…
Allow: /diveshop/index.php?main_page=popup_image&pID=*
Allow: /diveshop/index.php?main_page=popup_image_additional*
Disallow: /diveshop/index.php?main_page=popup_image*

That worked.

Google is still insisting that it wants to index pages like
https://www.scubadoctor.com.au/diveshop/index.php?main_page=product_reviews_write&products_id=5692
even though they're blocked by robots.txt

I can't see any reason to allow product_reviews_write pages to be indexed. It's meaningless to index them. It's pointless given the bot and user will just see a login page.

On each product page there are two links in the reviews section like this.

The plan is to add rel="noindex" to those two links. Then maybe Google will get the message and stop complaining.

Zen Cart 1.5.5e, PHP 5.3.29 MySQL 5.5.42

17 Feb 2021, 5:12 AM
#2
drbyte avatar

drbyte

Sensei

Join Date:
Jan 2004
Posts:
63,515
Plugin Contributions:
176

Re: Indexed, though blocked by robots.txt

lloyd_borrett:

The plan is to add rel="noindex" to those two links. Then maybe Google will get the message and stop complaining.

It should already be setting noindex,nofollow on those pages, as per:
https://github.com/zencart/zencart/blob/7ae05632fe371150e5a6b05f90933208aaa1d4b3/includes/languages/english/meta_tags.php#L63-L65

.
Zen Cart - putting the dream of business ownership within reach of anyone!
Donate to: DrByte directly or to the Zen Cart team as a whole

Remember: Any code suggestions you see here are merely suggestions. You assume full responsibility for your use of any such suggestions, including any impact ANY alterations you make to your site may have on your PCI compliance.
Furthermore, any advice you see here about PCI matters is merely an opinion, and should not be relied upon as "official". Official PCI information should be obtained from the PCI Security Council directly or from one of their authorized Assessors.

17 Feb 2021, 1:04 PM
#4
lloyd_borrett avatar

lloyd_borrett

Zen Follower

Join Date:
Mar 2006
Location:
Rosebud, Victoria, Australia
Posts:
308
Plugin Contributions:
2

Re: Indexed, though blocked by robots.txt

The bot will never get to the product_reviews_write pages to see the meta tags there. It will be taken to a login page and get no further.

The login page has
<meta name="robots" content="noindex, nofollow" />

Yet still Google insists it doesn't want to be blocked.

Zen Cart 1.5.5e, PHP 5.3.29 MySQL 5.5.42