URGENT :: Google Indexing website even though I have Robot.txt to Disallow!

Hi all,

I created a couple of Demo websites that I have to put links on them in my original store for others to see, however I DON'T want them indexed at all!!

I added these to the robot.txt which I placed in my website root directory adn store directory...

User-agent: *
Disallow: /cgi-bin/
Disallow: /zenid=
Disallow: /index.php?main_page=popup_image
Disallow: /images/
Disallow: /test1/
Disallow: /test2/

Test1 & Test2 are the test zen-cart sites I don't want indexed!

They are still however getting indexed, how do I quickly stop them?

Any quick help would be greatly appreciated.

Best Regards,

What do you mean by "indexed"?

Are the URLs for those pages being shown along with page title and a snippet of text taken from the meta description or page content?

If so, then there is a problem that your robots.txt contains non-valid paths to disallow spidering, compared with the paths actually used by your store.

OR

Are the URLs for those pages being shown as URL-only entries, i.e. without a title and without a snippet, in the search results?

If so, then that is the correct behaviour. The robots.txt disallow directive stops search engines from spidering the page, therefore they don't see any of the page content, but it does not stop them from listing the URLs as just URLs when they find references to them on other sites, or on other pages of the same site.

Hi,
Thank you for your reply.

When I type my website in google it comes up with different pages, and the Test websites are one of them. Yes, I can see the main page title and meta tags snippet etc for the test page as well and can click on the cache page to see it all.

I have a HTML page which is the main website... for example... www. mydomain.com, then I have the demo websites which are located on www. mydomain.com/test1, www. mydomain.com/test2 etc..

I have the robot.txt in my root directory and and in the test zen-cart directories.

What do you think might be the problem? I already included what I have in the robot.txt in my previous post. Any idea?

Thank you once again.

Best Regards,