世界著名搜索公司的爬虫清单

AbachoBOT=Abacho.com
abcdatos_botlink=Abcdatos.com
http://www.abcdatos.com/botlink/=Abcdatos.com
AESOP_com_SpiderMan=Aesop.com
ah-ha.com crawler (crawler@ah-ha.com)=ah-ha.com
ia_archiver=Archive.org
Scooter=Altavista.com
Mercator=Altavista.com
Scooter2_Mercator_3-1.0=Altavista.com
roach.smo.av.com-1.0=Altavista.com
Tv_Merc_resh_26_1_D-1.0=Altavista.com
AltaVista-Intranet=Altavista.co.uk
jan.gelin@av.com=Altavista.co.uk
FAST-WebCrawler=alltheweb.com
crawler@fast.no=alltheweb.com
Acoon Robot=acoon.de
antibot=antisearch.net
Atomz=atomz.com
Buscaplus Robi=buscaplus.com
CanSeek/=canseek.ca
support@canseek.ca=canseek.ca
ChristCRAWLER=christcrawler.com
Crawler=crawler.de
admin@crawler.de=crawler.de
DaAdLe.com ROBOT/=daadle.com
RaBot=daum.net
Agent-admin/=daum.net
phortse@hanmail.net=daum.net
contact/jylee@kies.co.kr=kies.co.kr
DeepIndex=deepindex.com
DittoSpyder=ditto.com
Jack=domanova.co.uk
Speedy Spider=entireweb.com
ArchitextSpider=excite.com
ArchitectSpider=excite.com
Arachnoidea=euroseek.net
arachnoidea@euroseek.net=euroseek.net
EZResult=ezresults.com
Fast PartnerSite Crawler=fastsearch.net
FAST Data Search Crawler=fastsearch.net
KIT-Fireball=fireball.de
FyberSearch=fybersearch.com
GalaxyBot=galaxy.com
geckobot=geckobot.com
GenCrawler=gendoor.com
GeonaBot=geona.com
Googlebot=Google.com
googlebot@googlebot.com=Google.com
google=Google.com
moget/2.0=goo.ne.jp
moget@goo.ne.jp=goo.ne.jp
Aranha=girafa.com
Slurp.so/1.0=Yahoo
slurp@inktomi.com=Yahoo
Slurp/2.0j=Yahoo
www.inktomisearch.com=Yahoo
Slurp/2.0-KiteHourly=Yahoo
Slurp/2.0-OwlWeekly=Yahoo
spider@aeneid.com=Yahoo
Slurp/3.0-AU=Yahoo
Toutatis 2.5-2=hoppa.com
Hubater=hubat.com
IlTrovatore-Setaccio=iltrovatore.it
IncyWincy=incywincy.com
UltraSeek=infoseek.com
InfoSeek Sidewinder=infoseek.com
Mole2/1.0=intags.de
webmaster@intags.de=intags.de
MP3Bot=mp3bot.de
C-PBWF-ip3000.com-crawler=ip3000.com
ip3000.com-crawler=ip3000.com
kuloko-bot/0.2=kuloko.com
LNSpiderguy=lexis-nexis.com
NetResearchServer=look.com
MantraAgent=looksmart.com
NetResearchServer=loopimprovements.com
Lycos_Spider_(T-Rex)=lycos.com
JoocerBot=joocer.com
HenryTheMiragoRobot=mirago.co.uk
mozDex/=mozdex.com
MSNBOT/0.1=MSN
Gulliver=northernlight.com
ObjectsSearch/0.01=objectssearch.com
PicoSearch/=picosearch.com
PJspider=portaljuice.com
DIIbot=powerinter.net
nttdirectory_robot=navi.ocn.ne.jp
super-robot@super.navi.ocn.ne.jp=navi.ocn.ne.jp
griffon=super.navi.ocn.ne.jp
griffon@super.navi.ocn.ne.jp=super.navi.ocn.ne.jp
Spider/maxbot.com=maxbot.com
admin@maxbot.com=maxbot.com
gazz/1.0=Unknown Spider
gazz@nttrd.com=Unknown Spider
NationalDirectory-SuperSpider=nationaldirectory.com
dloader(NaverRobot)/=naver.com
dumrobo(NaverRobot)/=naver.com
Openfind piranha=openfind.com
Shark=openfind.com
robot-response@openfind.com.tw=openfind.com.tw
Openbot/=openfind.com.tw
psbot=picsearch.org
CrawlerBoy=pinpoint.com
ip3000.com=petersnews.com
AlkalineBOT=AlkalineBOT
Fluffy the spider=searchhippo.com
info@searchhippo.com=searchhippo.com
Scrubby/=scrubtheweb.com
asterias=singingfish.com
speedfind ramBot xtreme=speedfind.de
Kototoi/0.1=s.u-tokyo.ac.jp
Searchspider/=searchspider.com