Disabled categories still found by Googlebot (and showing up in searches)

[gernot]

Hi all,

ZenCart 1.5.6c, MariaDB 10.3.20, PHP 7.3.11
https://www.proteawines.jp/shop/

Apart from the production products and categories, I have dummy products on my site, as well as other products and categories that I do not want to make public at the moment. So these categories and products are all disabled in the Admin categories/products screen (red icons for status instead of green).

However, Google Webmaster Tools tells me the links to the categories can still be found, when I click the URL indeed a page for the category comes up, although it contains the text that there are no products available. So Zen Cart is still aware of the URL and presents it to visitors even though they have no direct access to it through menus or otherwise from inside Zen Cart.

Here is an example:
https://www.proteawines.jp/shop/inde...73&language=en

Is this expected, and if so what is the best solution? These are not out of stock products so I expect I can merely delete the categories and products from Zen Cart, although this is not what I want to do (I want to keep test categories and products around as well).

Best regards,
Gernot Hassenpflug

[mc12345678]

Well, first of all, if a link on the Internet exists or perhaps previously existed to a location, then it is locateable by such search engines. In fact, by posting the above link, that has been added to indicate that it is a location to link. This is more of a factor if an existing link has been modified to such that it previously existed and now doesn't.

As far as attempting to make it not show, I can't recall if a disabled category applies the associated noindex, nofollow meta tags or not, but even if so, the search engine may need to be asked to inspect the page(s) in order for this to happen. Then there is also any sitemap that may still point to it.

So there are a lot of ways that something can remain visible to search engines including if they choose to just do a numerical search once a pattern has been identified. That last part is likely a stretch. I've looked at the page and don't see the noindex, nofollow indicators. Can take a look at includes/languages/YOUR_LANGUAGE/YOUR_TEMPLATE/mmeta_tags.php to see what may be necessary to have robots skip that/those pages, though it will take some time for them to be removed if not done in the associated webmaster tools.

[gernot]

Hi mc123456789,
Thank you for the reply. Good to know there are several things to consider here. First line of enquiry is to check what Zen Cart's responsibility and expected behaviour is, and thereafter to work on the external parts like Webmaster Tools.

So, for firstly for Zen Cart:

I gather from your reply that the fact that disabled pages are accessible, i.e., they are dynamically created if accessed at their URL, is expected behaviour. That is quite surprising to me, as I would have expected (without knowing anything else about the logic involved) that these pages simply cease to exist, and users would be presented with the define_page_not_found (which I spent quite a bit of effort on). If that is indeed expected behaviour, I need to keep that in mind, and also explain how this works to my partner who is handling the administration of products but is only just getting into understanding SEO issues (about 1 foot behind me, heh).

Next, I see that I do not have a custom meta_tags.php file, so I am using the default one from english/meta_tags.php (I can post it if needed, but it is untouched by me from the original in 1.5.6c). I really had no idea I might need to touch this file, so that is news and something more to learn. Forgive me for saying this as obviously Zen Cart is open-source and all that, but it seems to me I have stumbled into something that is either incomplete or tricky to handle safely. The good thing of course is that this thread may help other people with similar problems.
I'm not sure what I need to think about regarding the meta_tags.php file, most of the defines are empty strings. I'll search for some related threads on meta_tags.php before I edit the (new override) file.

Next, the sitemap. Yes, I am using the SitemapXML v3.9.6 plugin, and looking at the files it generated they are from July 2019, although I opened the store officially on December 1st, 2019! So the sitemap is totally out of whack with the actual store. I did not realize that this is not updated at each edit or on a regular basis. I should note though that I have not uploaded this sitemap to Google Search Console (sorry, this is what Webmaster Tools is now known as).
I will try to figure out in the admin part of the shop how to update the sitemap, and how not to forget to do this when necessary. Oops!
I should also check whether an updated sitemap includes these problematic pages or not.

Lastly, for Google Search Console:

I will see if I can apply to have problematic links removed, and additionally upload the updated sitemap.
Not sure what else I can do on Google, but after any changes on the site I can ask Google to rescan pages that gave problems.

Hope the above is a reasonable approach to go forward. Main blocker at the moment is not knowing what the meta_tags.php file changes should be.

[gernot]

Well, one bit of quick good news: Under Admin/Tools/SitemapXML I could regenerate the sitemap files, and these correctly only include the active categories (for each language). So those files are now correct on my site.
And now I will remember this is manual process!

[mc12345678]

Well, one bit of quick good news: Under Admin/Tools/SitemapXML I could regenerate the sitemap files, and these correctly only include the active categories (for each language). So those files are now correct on my site.
And now I will remember this is manual process!

It can be made an automatic process through a cron job. There is some guidance on the sitemap page in the admin. Lower right corner if I remember correctly.

As to having notified google or not, the plugin generates normal expected files for a site that has a sitemap that Google also looks for, so notification isn't necessarily necessary if/when it looks for one, but certainly helps to identify the existence.

[mc12345678]

I went back to look at the meta_tags file and realized that because this issue is specific to a category and not just one "page" that addressing the presence of noindex, nofollow is a little more than just placing something in the language file. In fact would pretty much take a different way to do this.

There appear to be a number of places where code could be added to turn on the flag $robotsNoIndex, setting it to true. I didn't look back to how far/long that variable has been available, but it is in 1.5.6. What you would want to do is at some point in preparing to show just the category (assuming that you would want to hide the category, but perhaps not the product within it), inspect the final category in the category list/array and if that is disabled, then to set $robotsNoIndex to true. Alternatively if you wanted to allow some, but not all such categories, then to identify them by categories_id and if the current display is that category, to disable it by setting to true... There is already an init_includes file (dealing with breadcrumbs) that does this if one of the categories does not exist (by category name not existing which is different than being empty). That could be used as a sort of template to accomplish the above... Again though only looking at the category that is at the end of $_GET['cPath'], though it could be that you have a disabled category at the top of a tree and you want all other categories under that tree branch to also be excluded... all in what you are wanting to do/get. :)

[gernot]

Hi mc123456789,
Thanks for the detailed reply. Interesting. A quick check shows that robotsNoIndex is in the following files on my system:

• includes/init_includes/init_add_crumbs.php
• includes/modules/pages/index/header_php.php
• includes/modules/pages/product_info/header_php.php
• includes/templates/<CUSTOM>/common/html_header.php <- my own template override using responsive_classic as a base

I will take up your advice and see how the breadcrumb logic can be applied to categories. Need to devour that code for a while and understand how the category pages are created.
I realize now this might apply also the manufacturer pages, but have not verified that on Google yet.

In case it simplifies things (and helps others looking later), I have only one level of category in my own products, but also all the dummy categories and products. All products inside categories that are disabled, are also disabled. So there is no visible category or product underneath a disabled one.

[gernot]

Some informational updates:

Looking at Google search results, I confirmed that not only categories but also disabled products are listed in search results.
I do not know how Google found these: it is likely in my view that Google found my old sitemap and used the links listed there (now corrected with an updated sitemap).

Google Search Console instructions (https://support.google.com/webmaster.../1663419?hl=en) indicate that unwanted URLs can be temporarily (up to 90 days) removed from the search results, but will re-appear after that (presumably if Google still finds them according to their way of crawling).
The only ways to get rid of pages permanently is by making sure the server returns a 404 or 401 HTTP status code, by blocking access with for example a password, or (not as secure a method) by having a noindex meta tag on that page.

So for Zen Cart dummy data products and categories on my site, I will go ahead and remove the URLs temporarily in Google Search Console and delete the dummy data (server will return a page not found HTTP 404 status code in that case). I also need to delete manufacturers.

For non-dummy data (currently only test products, no categories) that I have disabled, I will also go ahead and remove the URLs temporarily in Google Search Console, and then try to figure out how to add the noindex meta tag to the relevant pages: categories, products, manufacturers (not sure there are other ways disabled items can be searchable, maybe by product type as well, this will become clear as I go through the URLs I find).

[twitchtoo]

Have a look at Twitch Restrict - it addresses some security flaws in the stock Zen Cart.

https://www.zen-cart.com/downloads.php?do=file&id=1819

Since nothing has changed in a stock Zen Cart to prevent this - since 2014 when I built the code - as you can still manually get to anything in the site if you guess the name or number.

One of the endpoints is a Zen core file that allows predictive address access which the Twitch Restrict code stops.

Then spiders and search bots can't see anything that's 'restricted'. There's no reason why the source code can't be adapted to 1.5x for all pages as it has been working since 2014 as is (for categories) :).

[gernot]

Hello twitchtoo,
Many thanks for the helpful pointer, had a very brief look at the package files, pretty clear it is quite an impressive amount of work there. Admin side looks doable fairly easily, the front side a few more files to go through, and I will need to do this slowly as I have customized my cart to handle multiple languages (Japanese).

From a functional point of view this looks pretty much what I am looking for, still need to spend some time reading the support thread, understanding the implications and how to operate this plugin (to explain to my partner also who will manage the shop admin and products generally).

Thanks very much, maybe others will chime in here further, but this sounds like a better choice than building something from scratch (which might be easier for experienced Zen Cart programmers).