Spider- General Info

Is there a general FYI postor thread about Spiders? (i.e. how often they will, on average, visit your site; how much bandwidth they can take; if you can limit them, and if so, how.)

Now that our store is up and running and I'm checking out the "Who's Online" and Website stats through our host I notice a lot of Spider activity.

I searched the forum and have read a few threads about how to prevent them, but understand that it has advantages and disadvantages.

If not, could someone with some experience please post a brief summary? Would really appreciate it. (Should be a sticky somewhere- if I missed it I apologize)

You don't ever want to stop spiders coming in your site They will list your website on major search engines which will improve sells. There is a mod though that blocks spiders trying to steal emails in the download section though.
Sometimes you can have alot of spiders in your site, but I don't know how much bandwidth they use it's not much I'm sure. If you block spiders from all your pages you won't be listed in search engines, and thats not a good thing if you like getting new customers.

Spiders mostly = good :)

shackle:

Spiders mostly = good :)That was my understanding, but now being on the other side of running a website, (and paying for the bandwidth) I was just wondering if there is such a thing as too much spidering. Isn't there a CSS/HTML command that will limit how often they visit your site? Does it tend to slow down after some initial visits or is it a constant thing?

David,
you have plenty of bandwidth,
let the spiders eat at your pages till their hearts content, it will do nothing but help you,

but you may want to take shackles advise about the download that blocks the spiders from obtaining email addresses

other than that let em go

Guess I've been kinda surprised as to how long it takes them, after all they are computers...would think they'd be on and off pretty quickly. Glad to know I don't have to worry too much about them.

the longer google is on your page, the more info its digesting,

There are a lot of bad bots out there. You block them by .htaccess not robots.txt. Bad bots ignore robots.txt.

Of course you want Google, Slurp (Yahoo), MSN, and the other major search engines.

The best way to get bots to visit is have good site map, good internal linking, lots of relevant text, and new/changing content. If you do this, the big bots will visit often. I have had Slurp and Google visit sites 100s of times a day.

Bad bots are quite often email harvesters or even worse.

This is just a partial list of bad bots out there:

http://www.kloth.net/internet/badbots.php

There are over 11,000 known bad bots.

Newbie question here - have been watching the 'who's online' in my new Zen cart with interest - and seen Googlebot and inktomisearch in there a lot - which thanks to this thread I had realised was a GOOD THING!

Yesterday we went 'live' and actually put stock quantities in (previously everything was shown as 'out of stock' whilst we loaded the store.
Today I got all excited as someone had placed a large quantity of items in their shopping cart (c'mon - you must remember that feeling :lol:) until I realised that it was a googlebot.

My question is - do they usually go shopping? I thought they just looked?