Robots Text

[Bruce1952]

Hi All,

Running 1.58a with php 8.1

I want to find out about robot text as I am only showing one spider in the whos-online list on my site which is this one:

Host: ec2-47-128-125-172.ap-southeast-1.compute.amazonaws.com
User Agent: Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; [email protected])

This is what is in the present robot text file:

User-agent: Googlebot
Disallow: /nogooglebot/

User-agent: *
Allow: /

Sitemap: https://www.outdoorking.com.au/sitemap.xml

The question is should I do any changes to get other search engines like bing etc to search the site or will it slow the whole site down with other crawlers searching.

Any assistance would be appreciated as I am not sure what I should be doing with this file.

[Shop Suey]

At best you don't do things you aren't sure how they exactly work.

If you had serious scammers than you could follow https://www.zen-cart.com/showthread....ighlight=abuse

[Bruce1952]

At best you don't do things you aren't sure how they exactly work.

If you had serious scammers than you could follow https://www.zen-cart.com/showthread....ighlight=abuse

That is why I asked the question here. I will try again because I don't think people understand what I said.

1. I am only showing one spider in the list of the admin area.

2. I have shown what is in the robot text file so people can see what is in the file at present.

3. I wanted to see if anyone could point me in the right direction what should be in the robots text file so that I can increase searches without to many bad spiders.

If replies can't assist then don't reply with stuff that is not of any use.

[mprough]

That is why I asked the question here. I will try again because I don't think people understand what I said.

1. I am only showing one spider in the list of the admin area.

2. I have shown what is in the robot text file so people can see what is in the file at present.

3. I wanted to see if anyone could point me in the right direction what should be in the robots text file so that I can increase searches without to many bad spiders.

If replies can't assist then don't reply with stuff that is not of any use.

The robots.txt file does not prevent any bots from your site. It is a list of instructions for well behaving bots to hopefully abide by.
It also doesn't really help your search engine results, except possibly for the sitemap inclusion.

Your robots.txt isn't valid. Do not include the 'Allow' directive before any 'Disallow', despite Google and Bing supporting it. Also the area you are trying to block... If this is because some pages/resources in this folder got indexed, then remove them in webmaster's tools. Otherwise you should never publicize areas you don't want people/bots in. Google will not remove an indexed page/resource that is only blocked in robots.txt, you must also supply a no-index tag or remove it in webmaster's tools.

The bang for your buck move to improve your search engine visibility would be to work on your website textual content & speed. =)

~Melanie

[Shop Suey]

Exactly, robots.txt is just an instruction file. Spiders who don't care of your instruction will just disregard it.
It seems you don't understand why I send you the link.

There are lots of information at the web how to write proper syntax in your robots.txt.
And also information of how crawlers changed on followin your instructions for nofollow and noindex for severeal years.

Again you shouldn't fiddle around with those settings if didn't know exactly what you are doing. It likely won't do things better. Instead make your shop more beautiful.

And finally this is not Zen Cart related.