1. Spider Names
Google spider: googlebot
Baidu spider: baiduspider
Sogou spider: sogou spider
Soso spider: Sosospider
Yahoo spider: slurp
Alexa spider: ia_archiver
MSN spider: msnbot
Altavista spider: scooter
Lycos spider: lycos_spider_(t-rex)
Alltheweb spider: fast-webcrawler
Inktomi spider: slurp
Youdao spider: YodaoBot and OutfoxBot
Retu spider: Adminrtspider
That’s just a representative list, so don’t be surprised when you see them in your logs — and if you want to block them from crawling your site, you can.
2. Spider Habits, Summarized
Google spider: doesn’t crawl much, but loves to index.
Baidu spider: cautious about crawling, even more cautious about indexing.
Soso spider: loves crawling images, and often gets stuck wandering around dynamic URLs.
Yahoo spider: follows the rules, always crawling robots.txt first.
I’ll leave the others out.
3. How Well They Honor robots.txt
Covering all of them isn’t realistic, so let’s take Disallow: /*?* (blocking dynamic pages) as the example.
Google: once you write the rule it stops crawling, and Google Webmaster Tools lists which URLs it wanted to crawl but was blocked from.
Baidu: crawls rarely after the rule is written, but still crawls occasionally. I believe it works, because the frequency drops — several times a day before, once every few days now.
Sogou: basically disobedient. I’m not sure whether it doesn’t understand the rule or just ignores it; it doesn’t ignore it completely — it strips the question mark off dynamic URLs and crawls them anyway, a huge swath at a time. What it expects to get out of that, I don’t know.
Soso and Yahoo seem similar; both feel fairly effective, with no crawl traces after being blocked.