greg_beyer:
Hello, I'm trying to figure out why Goog is not indexing our categories, as shown in the Google Search Console. This despite that I have a sitemap.txt in my webroot, and it's successfully submitted to Google (no errors).
There is not robots.txt to lead search engines to your sitemap. a txt file is adequate, the the protocol calls for a proper sitemap.xml file. Your current sitemap.txt file is not complete and only contains 22 urls. I use xml-sitemaps.com instead of the plugin here because it actually crawls and thus identifies issues that would otherwise cause issues for the bots.
greg_beyer:
So I'm wondering if anyone else has experienced this, and what I can do to resolve it.
Does it have something to do with categories (and all Z-C product pages) not being actual static pages, but rather generated on-the-fly out of the database? But then again, any large website (think media sites, Amazon) are dymanic db driven, so I don't see how that could be the issue.
I believe you are mistaken paste this in Google without the quotes "site:australianbakerycafe.com +-c-". You will see 8 of your 12 category pages are in fact indexed.
Search for "site:australianbakerycafe.com" too see all 706 pages indexed.
It was here that I found you changed your urls. Unfortunately, it is a huge mistake to do this on a live website. You basically have to wait now for Google to trust your site again. It will be a few months minimum up to a year or so for any rank to return. Do NOT turn off the url rewriter it will only BE LONGER in the sandbox. I would get a proper robotx.txt Like below and a proper sitemap. Then add new textual content on a regular basis, structured data and wait.
Some questions:
How long has your site been live?
Was your site live already when you added the url rewriter? If so when did you add it
robots.txt
# Robots.txt file for https:/domain.com/
# Updated 09/06/2020
Sitemap: https://domain.com/sitemap.xml
User-agent: *
Disallow: /cgi-bin/
This is FULLY adequate DO NOT block any of your urls in robots.txt this DOES NOT block, nor prevent indexing, just crawling. To prevent indexing you MUST use a noindex tag. Do not block folders like your admin. These would not be discovered on crawl and adding it to robots.txt is like giving the hackers a map.
greg_beyer:
Our site is australianbakery.com. It re-directs to australianbakerycafe.com
Thanks
Why would you do that? That is a serious problem with regard to the search engines. Did you recently move your site?
~Melanie
