User-agent: Gagglebot Disallow: / User-agent: Naamah 1.0.1/Blogbot (http://blogbot.de/) Disallow: / User-agent: */Nutch-0.9-dev Disallow: / User-agent: +SitiDi.net/SitiDiBot/1.0 (+Have Good Day) Disallow: / User-agent: -DIE-KRAEHE- META-SEARCH-ENGINE/1.1 http://www.die-kraehe.de Disallow: / User-agent: 192.comAgent Disallow: / User-agent: 4anything.com LinkChecker v2.0 Disallow: / User-agent: 8484 Boston Project v 1.0 Disallow: / User-agent: :robot/1.0 (linux) ( admin e-mail: undefined http://www.neofonie.de/loesungen/search/robot.html ) Disallow: / User-agent: A-Online Search Disallow: / User-agent: A1 Sitemap Generator/1.0 (+http://www.micro-sys.dk/products/sitemap-generator/) miggibot/2006.01.24 Disallow: / User-agent: ABCdatos BotLink/5.xx.xxx#BBL Disallow: / User-agent: AESOP_com_SpiderMan Disallow: / User-agent: AIBOT/2.1 By +(www.21seek.com A Real artificial intelligence search engine China) Disallow: / User-agent: ASAHA Search Engine Turkey V.001 (http://www.asaha.com/) Disallow: / User-agent: ASPSeek/1.2.5 Disallow: / User-agent: ASPSeek/1.2.x Disallow: / User-agent: ASPSeek/1.2.xa Disallow: / User-agent: ASPSeek/1.2.xxpre Disallow: / User-agent: ASPseek/1.2.9d Disallow: / User-agent: ASPseek/1.2.xx Disallow: / User-agent: ASSORT/0.10 Disallow: / User-agent: AV Fetch 1.0 Disallow: / User-agent: AVSearch-1.0(peter.turney@nrc.ca) Disallow: / User-agent: AVSearch-3.0(AltaVista/AVC) Disallow: / User-agent: AbachoBOT Disallow: / User-agent: AbachoBOT (Mozilla compatible) Disallow: / User-agent: Aberja Checkomat Disallow: / User-agent: About/0.1libwww-perl/5.47 Disallow: / User-agent: Accelatech RSSCrawler/0.4 Disallow: / User-agent: Accoona-AI-Agent/1.1.1 (crawler at accoona dot com) Disallow: / User-agent: Accoona-AI-Agent/1.1.2 (aicrawler at accoonabot dot com) Disallow: / User-agent: Ack (http://www.ackerm.com/) Disallow: / User-agent: AcoiRobot Disallow: / User-agent: Acoon Robot v1.50.001 Disallow: / User-agent: Acoon Robot v1.52 (http://www.acoon.de) Disallow: / User-agent: Acoon-Robot 4.0.x.[xx] (http://www.acoon.de) Disallow: / User-agent: Acoon-Robot v3.xx (http://www.acoon.de and http://www.acoon.com) Disallow: / User-agent: Acorn/Nutch-0.9 (Non-Profit Search Engine; acorn.isara.org; acorn at isara dot org) Disallow: / User-agent: Agent-SharewarePlazaFileCheckBot/2.0+(+http://www.SharewarePlaza.com) Disallow: / User-agent: AgentName/0.1 libwww-perl/5.48 Disallow: / User-agent: Aladin/3.324 Disallow: / User-agent: Aleksika Spider/1.0 (+http://www.aleksika.com/) Disallow: / User-agent: AlkalineBOT/1.3 Disallow: / User-agent: AlkalineBOT/1.4 (1.4.0326.0 RTM) Disallow: / User-agent: Allesklar/0.1 libwww-perl/5.46 Disallow: / User-agent: AltaVista Intranet V2.0 AVS EVAL search@freeit.com Disallow: / User-agent: AltaVista Intranet V2.0 Compaq Altavista Eval sveand@altavista.net Disallow: / User-agent: AltaVista Intranet V2.0 evreka.com crawler@evreka.com Disallow: / User-agent: AltaVista V2.0B crawler@evreka.com Disallow: / User-agent: AmfibiBOT Disallow: / User-agent: Amfibibot/0.06 (Amfibi Web Search; http://www.amfibi.com; agent@amfibi.com) Disallow: / User-agent: Amfibibot/0.07 (Amfibi Robot; http://www.amfibi.com; agent@amfibi.com) Disallow: / User-agent: AnnoMille spider 0.1 alpha - http://www.annomille.it Disallow: / User-agent: AnswerBus (http://www.answerbus.com/) Disallow: / User-agent: AnzwersCrawl/2.0 (anzwerscrawl@anzwers.com.au;Engine) Disallow: / User-agent: Apexoo Spider 1.x Disallow: / User-agent: Aport Disallow: / User-agent: ArabyBot (compatible; Mozilla/5.0; GoogleBot; FAST Crawler 6.4; http://www.araby.com;) Disallow: / User-agent: ArachBot Disallow: / User-agent: Arachnoidea (arachnoidea@euroseek.com) Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: Arikus_Spider Disallow: / User-agent: Arquivo-web-crawler (compatible; heritrix/1.12.1 +http://arquivo-web.fccn.pt) Disallow: / User-agent: Asahina-Antenna/1.x Disallow: / User-agent: Asahina-Antenna/1.x (libhina.pl/x.x ; libtime.pl/x.x) Disallow: / User-agent: AskAboutOil/0.06-rcp (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@askaboutoil.com) Disallow: / User-agent: AtlocalBot/1.1 +(http://www.atlocal.com/local-web-site-owner.html) Disallow: / User-agent: Atomic_Email_Hunter/4.0 Disallow: / User-agent: Atomz/1.0 Disallow: / User-agent: Attentio/Nutch-0.9-dev (Attentio's beta blog crawler; www.attentio.com; info@attentio.com) Disallow: / User-agent: AxmoRobot - Crawling your site for better indexing on www.axmo.com search engine. Disallow: / User-agent: BDFetch Disallow: / User-agent: BDNcentral Crawler v2.3 [en] (http://www.bdncentral.com/robot.html) (X11; I; Linux 2.0.44 i686) Disallow: / User-agent: BIGLOTRON (Beta 2;GNU/Linux) Disallow: / User-agent: BPImageWalker/2.0 (www.bdbrandprotect.com) Disallow: / User-agent: BSDSeek/1.0 Disallow: / User-agent: BTbot/0.x (+http://www.btbot.com/btbot.html) Disallow: / User-agent: BaboomBot/1.x.x (+http://www.baboom.us) Disallow: / User-agent: BaiDuSpider Disallow: / User-agent: BaiduImagespider+(+http://www.baidu.jp/search/s308.html) Disallow: / User-agent: Baiduspider+(+http://help.baidu.jp/system/05.html) Disallow: / User-agent: Baiduspider+(+http://www.baidu.com/search/spider.htm) Disallow: / User-agent: Baiduspider+(+http://www.baidu.com/search/spider_jp.html) Disallow: / User-agent: Balihoo/Nutch-1.0-dev (Crawler for Balihoo.com search engine - obeys robots.txt and robots meta tags ; http://balihoo.com/index.aspx; robot at balihoo dot com) Disallow: / User-agent: BarraHomeCrawler (albertof@barrahome.org) Disallow: / User-agent: BeamMachine/0.5 (dead link remover of www.beammachine.net) Disallow: / User-agent: BebopBot/2.5.1 ( crawler http://www.apassion4jazz.net/bebopbot.html ) Disallow: / User-agent: BigCliqueBOT/1.03-dev (bigclicbot; http://www.bigclique.com; bot@bigclique.com) Disallow: / User-agent: Bigsearch.ca/Nutch-x.x-dev (Bigsearch.ca Internet Spider; http://www.bigsearch.ca/; info@enhancededge.com) Disallow: / User-agent: BilgiBetaBot/0.8-dev (bilgi.com (Beta) ; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org) Disallow: / User-agent: BilgiBot/1.0(beta) (http://www.bilgi.com/; bilgi at bilgi dot com) Disallow: / User-agent: Bitacle Robot (V:1.0;) (http://www.bitacle.com) Disallow: / User-agent: Bitacle bot/1.1 Disallow: / User-agent: BlackWidow Disallow: / User-agent: Blaiz-Bee/1.0 (+http://www.blaiz.net) Disallow: / User-agent: Blaiz-Bee/2.00.8222 (BE Internet Search Engine http://www.rawgrunt.com) Disallow: / User-agent: Blaiz-Bee/2.00.xxxx (+http://www.blaiz.net) Disallow: / User-agent: BlitzBOT@tricus.net Disallow: / User-agent: BlitzBOT@tricus.net (Mozilla compatible) Disallow: / User-agent: BlogBot/1.x Disallow: / User-agent: BlogPulseLive (support@blogpulse.com) Disallow: / User-agent: BlogSearch/1.x +http://www.icerocket.com/ Disallow: / User-agent: BlogVibeBot-v1.1 (spider@blogvibe.nl) Disallow: / User-agent: Bloglines Title Fetch/1.0 (http://www.bloglines.com) Disallow: / User-agent: Bloglines-Images/0.1 (http://www.bloglines.com) Disallow: / User-agent: Blogpulse (info@blogpulse.com) Disallow: / User-agent: BlogsNowBot, V 2.01 (+http://www.blogsnow.com/) Disallow: / User-agent: BlogzIce/1.0 (+http://icerocket.com; rhodes@icerocket.com) Disallow: / User-agent: BlogzIce/1.0 +http://www.icerocket.com/ Disallow: / User-agent: BloobyBot Disallow: / User-agent: Bloodhound/Nutch-0.9 (Testing Crawler for Research - obeys robots.txt and robots meta tags ; http://balihoo.com/index.aspx; robot at balihoo dot com) Disallow: / User-agent: BravoBrian SpiderEngine MarcoPolo Disallow: / User-agent: BruinBot (+http://webarchive.cs.ucla.edu/bruinbot.html) Disallow: / User-agent: BuildCMS crawler (http://www.buildcms.com/crawler) Disallow: / User-agent: BullsEye Disallow: / User-agent: BurstFindCrawler/1.1 (crawler.burstfind.com; http://crawler.burstfind.com; crawler@burstfind.com) Disallow: / User-agent: Buscaplus Robi/1.0 (http://www.buscaplus.com/robi/) Disallow: / User-agent: CCBot/1.0 (+http://www.commoncrawl.org/bot.html) Disallow: / User-agent: Cabot/Nutch-0.9 (Amfibi's web-crawling robot; http://www.amfibi.com/cabot/; agent@amfibi.com) Disallow: / User-agent: Cabot/Nutch-1.0-dev (Amfibi's web-crawling robot; http://www.amfibi.com/cabot/; agent@amfibi.com) Disallow: / User-agent: Carnegie_Mellon_University_Research_WebBOT-->PLEASE READ-->http://www.andrew.cmu.edu/~brgordon/webbot/index.html http://www.andrew.cmu.edu/~brgordon/webbot/index.html Disallow: / User-agent: Carnegie_Mellon_University_WebCrawler http://www.andrew.cmu.edu/~brgordon/webbot/index.html Disallow: / User-agent: Catall Spider Disallow: / User-agent: CazoodleBot/CazoodleBot-0.1 (CazoodleBot Crawler; http://www.cazoodle.com/cazoodlebot; cazoodlebot@cazoodle.com) Disallow: / User-agent: Ceramic Tile Installation Guide (http://www.floorstransformed.com) Disallow: / User-agent: China Local Browse 2.6 Disallow: / User-agent: ChristCRAWLER 2.0 Disallow: / User-agent: CipinetBot (http://www.cipinet.com/bot.html) Disallow: / User-agent: ClariaBot/1.0 Disallow: / User-agent: Claymont.com Disallow: / User-agent: CloakDetect/0.9 (+http://fulltext.seznam.cz/) Disallow: / User-agent: Clushbot/2.x (+http://www.clush.com/bot.html) Disallow: / User-agent: Clushbot/3.x-BinaryFury (+http://www.clush.com/bot.html) Disallow: / User-agent: Clushbot/3.xx-Ajax (+http://www.clush.com/bot.html) Disallow: / User-agent: Clushbot/3.xx-Hector (+http://www.clush.com/bot.html) Disallow: / User-agent: Clushbot/3.xx-Peleus (+http://www.clush.com/bot.html) Disallow: / User-agent: CoBITSProbe Disallow: / User-agent: Combine/2.0 http://combine.it.lth.se/ Disallow: / User-agent: Combine/3 http://combine.it.lth.se/ Disallow: / User-agent: Combine/x.0 Disallow: / User-agent: Computer_and_Automation_Research_Institute_Crawler crawler@ilab.sztaki.hu Disallow: / User-agent: Comrite/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org) Disallow: / User-agent: ContactBot/0.2 Disallow: / User-agent: ContentSmartz Disallow: / User-agent: Convera Internet Spider V6.x Disallow: / User-agent: ConveraCrawler/0.2 Disallow: / User-agent: ConveraCrawler/0.9d (+http://www.authoritativeweb.com/crawl) Disallow: / User-agent: ConveraMultiMediaCrawler/0.1 (+http://www.authoritativeweb.com/crawl) Disallow: / User-agent: CoolBot Disallow: / User-agent: CougarSearch/0.x (+http://www.cougarsearch.com/faq.shtml) Disallow: / User-agent: Covac TexAs Arachbot Disallow: / User-agent: Cowbot-0.1 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com) Disallow: / User-agent: Cowbot-0.1.x (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com) Disallow: / User-agent: CrawlConvera0.1 (CrawlConvera@yahoo.com) Disallow: / User-agent: Crawler (cometsearch@cometsystems.com) Disallow: / User-agent: Crawler V 0.2.x admin@crawler.de Disallow: / User-agent: Crawler admin@crawler.de Disallow: / User-agent: CrawlerBoy Pinpoint.com Disallow: / User-agent: Crawllybot/0.1 (Crawllybot; +http://www.crawlly.com; crawler@crawlly.com) Disallow: / User-agent: CreativeCommons/0.06-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net) Disallow: / User-agent: CrocCrawler vx.3 [en] (http://www.croccrawler.com) (X11; I; Linux 2.0.44 i686) Disallow: / User-agent: Cuasarbot/0.9b http://www.cuasar.com/spider_beta/ Disallow: / User-agent: CurryGuide SiteScan 1.1 Disallow: / User-agent: Custom Spider www.bisnisseek.com /1.0 Disallow: / User-agent: CyberPatrol SiteCat Webbot (http://www.cyberpatrol.com/cyberpatrolcrawler.asp) Disallow: / User-agent: CydralSpider/1.x (Cydral Web Image Search; http://www.cydral.com) Disallow: / User-agent: CydralSpider/3.0 (Cydral Image Search; http://www.cydral.com) Disallow: / User-agent: DBrowse 1.4b Disallow: / User-agent: DBrowse 1.4d Disallow: / User-agent: DIIbot/1.2 Disallow: / User-agent: DSurf15a 01 Disallow: / User-agent: DSurf15a 71 Disallow: / User-agent: DSurf15a 81 Disallow: / User-agent: DSurf15a VA Disallow: / User-agent: DTAAgent Disallow: / User-agent: DataCha0s/2.0 Disallow: / User-agent: DataFountains/DMOZ Downloader Disallow: / User-agent: DataFountains/DMOZ Feature Vector Corpus Creator (http://ivia.ucr.edu/useragents.shtml) Disallow: / User-agent: DataFountains/Dmoz Downloader (http://ivia.ucr.edu/useragents.shtml) Disallow: / User-agent: DataSpear/1.0 (Spider; http://www.dataspear.com/spider.html; spider@dataspear.com) Disallow: / User-agent: DataSpearSpiderBot/0.2 (DataSpear Spider Bot; http://dssb.dataspear.com/bot.html; dssb@dataspear.com) Disallow: / User-agent: DataparkSearch/4.47 (+http://dataparksearch.org/bot) Disallow: / User-agent: DataparkSearch/4.xx (http://www.dataparksearch.org/) Disallow: / User-agent: DatenBot( http://www.sicher-durchs-netz.de/bot.html) Disallow: / User-agent: DaviesBot/1.7 (www.wholeweb.net) Disallow: / User-agent: DeepIndex Disallow: / User-agent: DeepIndex ( http://www.zetbot.com ) Disallow: / User-agent: DeepIndex (www.en.deepindex.com) Disallow: / User-agent: DeepIndexer.ca Disallow: / User-agent: Demo Bot DOT 16b Disallow: / User-agent: Demo Bot Z 16b Disallow: / User-agent: Denmex websearch (http://search.denmex.com) Disallow: / User-agent: DiaGem/1.1 (http://www.skyrocket.gr.jp/diagem.html) Disallow: / User-agent: Diamond/x.0 Disallow: / User-agent: DiamondBot Disallow: / User-agent: DigOut4U Disallow: / User-agent: Digger/1.0 JDK/1.3.0rc3 Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DoCoMo/1.0/Nxxxi/c10 Disallow: / User-agent: DoCoMo/1.0/Nxxxi/c10/TB Disallow: / User-agent: DoCoMo/2.0 P900iV(c100;TB;W24H11) Disallow: / User-agent: DoCoMo/2.0 SH902i (compatible; Y!J-SRD/1.0; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-27.html) Disallow: / User-agent: DoCoMo/2.0/SO502i (compatible; Y!J-SRD/1.0; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-27.html) Disallow: / User-agent: Download-Tipp Linkcheck (http://download-tipp.de/) Disallow: / User-agent: Drecombot/1.0 (http://career.drecom.jp/bot.html) Disallow: / User-agent: DuckDuckBot/1.0; (+http://duckduckgo.com/duckduckbot.html) Disallow: / User-agent: Dumbot(version 0.1 beta - dumbfind.com) Disallow: / User-agent: Dumbot(version 0.1 beta - http://www.dumbfind.com/dumbot.html) Disallow: / User-agent: Dumbot(version 0.1 beta) Disallow: / User-agent: EARTHCOM.info/1.x [www.earthcom.info] Disallow: / User-agent: EARTHCOM.info/1.xbeta [www.earthcom.info] Disallow: / User-agent: EBrowse 1.4b Disallow: / User-agent: ELI/20070402:2.0 (DAUM RSS Robot, Daum Communications Corp.; +http://ws.daum.net/aboutkr.html) Disallow: / User-agent: EMPAS_ROBOT Disallow: / User-agent: ES.NET_Crawler/2.0 (http://search.innerprise.net/) Disallow: / User-agent: ESISmartSpider Disallow: / User-agent: ESurf15a 15 Disallow: / User-agent: EasyDL/3.xx Disallow: / User-agent: EasyDL/3.xx http://keywen.com/Encyclopedia/Bot Disallow: / User-agent: EchO!/2.0 Disallow: / User-agent: Educate Search VxB Disallow: / User-agent: EgotoBot/4.8 (+http://www.egoto.com/about.htm) Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailSpider Disallow: / User-agent: EmailWolf 1.00 Disallow: / User-agent: EnaBot/1.x (http://www.enaball.com/crawler.html) Disallow: / User-agent: Enfish Tracker Disallow: / User-agent: Enterprise_Search/1.0 Disallow: / User-agent: Enterprise_Search/1.0.xxx Disallow: / User-agent: Enterprise_Search/1.00.xxx;MSSQL (http://www.innerprise.net/es-spider.asp) Disallow: / User-agent: EroCrawler Disallow: / User-agent: EuripBot/0.x (+http://www.eurip.com) GetFile Disallow: / User-agent: EuripBot/0.x (+http://www.eurip.com) GetRobots Disallow: / User-agent: EuripBot/0.x (+http://www.eurip.com) PreCheck Disallow: / User-agent: Eurobot/1.0 (http://www.ayell.eu) Disallow: / User-agent: EvaalSE - bot@evaal.com Disallow: / User-agent: Everest-Vulcan Inc./0.1 (R&D project; host=e-1-24; http://everest.vulcan.com/crawlerhelp) Disallow: / User-agent: Everest-Vulcan Inc./0.1 (R&D project; http://everest.vulcan.com/crawlerhelp) Disallow: / User-agent: Exabot-Images/1.0 Disallow: / User-agent: Exabot-Test/1.0 Disallow: / User-agent: Exabot/2.0 Disallow: / User-agent: Exabot/3.0 Disallow: / User-agent: ExactSeek Crawler/0.1 Disallow: / User-agent: Exalead NG/MimeLive Client (convert/http/0.120) Disallow: / User-agent: Excalibur Internet Spider V6.5.4 Disallow: / User-agent: Execrawl/1.0 (Execrawl; http://www.execrawl.com/; bot@execrawl.com) Disallow: / User-agent: ExperimentalHenrytheMiragoRobot Disallow: / User-agent: Expired Domain Sleuth Disallow: / User-agent: ExtractorPro Disallow: / User-agent: EyeCatcher (Download-tipp.de)/1.0 Disallow: / User-agent: FAST Enterprise Crawler 6 (Experimental) Disallow: / User-agent: FAST Enterprise Crawler 6 / Scirus scirus-crawler@fast.no; http://www.scirus.com/srsapp/contactus/ Disallow: / User-agent: FAST Enterprise Crawler 6 used by Cobra Development (admin@fastsearch.com) Disallow: / User-agent: FAST Enterprise Crawler 6 used by Comperio AS (sts@comperio.no) Disallow: / User-agent: FAST Enterprise Crawler 6 used by FAST (FAST) Disallow: / User-agent: FAST Enterprise Crawler 6 used by Pages Jaunes (pvincent@pagesjaunes.fr) Disallow: / User-agent: FAST Enterprise Crawler 6 used by Sensis.com.au Web Crawler (search_comments\at\sensis\dot\com\dot\au) Disallow: / User-agent: FAST Enterprise Crawler 6 used by Singapore Press Holdings (crawler@sphsearch.sg) Disallow: / User-agent: FAST Enterprise Crawler/6 (www.fastsearch.com) Disallow: / User-agent: FAST Enterprise Crawler/6.4 (helpdesk at fast.no) Disallow: / User-agent: FAST FirstPage retriever (compatible; MSIE 5.5; Mozilla/4.0) Disallow: / User-agent: FAST MetaWeb Crawler (helpdesk at fastsearch dot com) Disallow: / User-agent: FAST-WebCrawler/2.2.10 (Multimedia Search) (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html) Disallow: / User-agent: FAST-WebCrawler/2.2.6 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html) Disallow: / User-agent: FAST-WebCrawler/2.2.7 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)http://www.fast.no Disallow: / User-agent: FAST-WebCrawler/2.2.8 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)http://www.fast.no Disallow: / User-agent: FAST-WebCrawler/3.2 test Disallow: / User-agent: FAST-WebCrawler/3.3 (crawler@fast.no; http://fast.no/support.php?c=faqs/crawler) Disallow: / User-agent: FAST-WebCrawler/3.4/Nirvana (crawler@fast.no; http://fast.no/support.php?c=faqs/crawler) Disallow: / User-agent: FAST-WebCrawler/3.4/PartnerSite (crawler@fast.no; http://fast.no/support.php?c=faqs/crawler) Disallow: / User-agent: FAST-WebCrawler/3.5 (atw-crawler at fast dot no; http://fast.no/support.php?c=faqs/crawler) Disallow: / User-agent: FAST-WebCrawler/3.6 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp) Disallow: / User-agent: FAST-WebCrawler/3.6/FirstPage (crawler@fast.no; http://fast.no/support.php?c=faqs/crawler) Disallow: / User-agent: FAST-WebCrawler/3.7 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp) Disallow: / User-agent: FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp) Disallow: / User-agent: FAST-WebCrawler/3.8 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp) Disallow: / User-agent: FAST-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; http://fast.no/support/crawler.asp) Disallow: / User-agent: FAST-WebCrawler/3.x Multimedia Disallow: / User-agent: FAST-WebCrawler/3.x Multimedia (mm dash crawler at fast dot no) Disallow: / User-agent: FFC Trap Door Spider Disallow: / User-agent: FSurf15a 01 Disallow: / User-agent: Factbot 1.09 (see http://www.factbites.com/webmasters.php) Disallow: / User-agent: Fast Crawler Gold Edition Disallow: / User-agent: Fast PartnerSite Crawler Disallow: / User-agent: FastBug http://www.ay-up.com Disallow: / User-agent: FastCrawler 3.0.1 (crawler@1klik.dk) Disallow: / User-agent: FastSearch Web Crawler for Verizon SuperPages (kevin.watters@fastsearch.com) Disallow: / User-agent: Favcollector/2.0 (info@favcollector.com http://www.favcollector.com/) Disallow: / User-agent: Faxobot/1.0 Disallow: / User-agent: Feed Seeker Bot (RSS Feed Seeker http://www.MyNewFavoriteThing.com/fsb.php) Disallow: / User-agent: Feed24.com Disallow: / User-agent: FeedChecker/0.01 Disallow: / User-agent: FeedHub FeedDiscovery/1.0 (http://www.feedhub.com) Disallow: / User-agent: FeedHub MetaDataFetcher/1.0 (http://www.feedhub.com) Disallow: / User-agent: Feedfetcher-Google; (+http://www.google.com/feedfetcher.html) Disallow: / User-agent: Feedjit Favicon Crawler 1.0 Disallow: / User-agent: Feedster Crawler/3.0; Feedster, Inc. Disallow: / User-agent: Felix - Mixcat Crawler (+http://mixcat.com) Disallow: / User-agent: Filangy/0.01-beta (Filangy; http://www.nutch.org/docs/en/bot.html; filangy-agent@filangy.com) Disallow: / User-agent: Filangy/1.0x (Filangy; http://www.filangy.com/filangyinfo.jsp?inc=robots.jsp; filangy-agent@filangy.com) Disallow: / User-agent: Filangy/1.0x (Filangy; http://www.nutch.org/docs/en/bot.html; filangy-agent@filangy.com) Disallow: / User-agent: Filtrbox/1.0 Disallow: / User-agent: FindAnISP.com_ISP_Finder_v99a Disallow: / User-agent: Findexa Crawler (http://www.findexa.no/gulesider/article26548.ece) Disallow: / User-agent: FineBot Disallow: / User-agent: Firefly/1.0 Disallow: / User-agent: Firefly/1.0 (compatible; Mozilla 4.0; MSIE 5.5) Disallow: / User-agent: Firefox (kastaneta03@hotmail.com) Disallow: / User-agent: Firefox_1.0.6 (kasparek@naparek.cz) Disallow: / User-agent: FirstGov.gov Search - POC:firstgov.webmasters@gsa.gov Disallow: / User-agent: Flapbot/0.7.2 (Flaptor Crawler; http://www.flaptor.com; crawler at flaptor period com) Disallow: / User-agent: Flexum spider Disallow: / User-agent: Flexum/2.0 Disallow: / User-agent: FlickBot 2.0 RPT-HTTPClient/0.3-3 Disallow: / User-agent: FocusedSampler/1.0 Disallow: / User-agent: Folkd.com Spider/0.1 beta 1 (www.folkd.com) Disallow: / User-agent: Fooky.com/ScorpionBot/ScoutOut; http://www.fooky.com/scorpionbots Disallow: / User-agent: Francis/1.0 (francis@neomo.de http://www.neomo.de/) Disallow: / User-agent: Franklin Locator 1.8 Disallow: / User-agent: FreeFind.com-SiteSearchEngine/1.0 (http://freefind.com; spiderinfo@freefind.com) Disallow: / User-agent: FreshNotes crawler< report problems to crawler-at-freshnotes-dot-com Disallow: / User-agent: Full Web Bot 0416B Disallow: / User-agent: Full Web Bot 0516B Disallow: / User-agent: Full Web Bot 2816B Disallow: / User-agent: FuseBulb.Com Disallow: / User-agent: FyberSpider (+http://www.fybersearch.com/fyberspider.php) Disallow: / User-agent: GAIS Robot/1.0B2 Disallow: / User-agent: GNODSPIDER (www.gnod.net) Disallow: / User-agent: GOFORITBOT ( http://www.goforit.com/about/ ) Disallow: / User-agent: Gaisbot/3.0 (indexer@gais.cs.ccu.edu.tw; http://gais.cs.ccu.edu.tw/robot.php) Disallow: / User-agent: Gaisbot/3.0+(robot06@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php) Disallow: / User-agent: GalaxyBot/1.0 (http://www.galaxy.com/galaxybot.html) Disallow: / User-agent: Gallent Search Spider v1.4 Robot 2 (http://robot.GallentSearch.com) Disallow: / User-agent: GammaSpider/1.0 Disallow: / User-agent: Generic Mobile Phone (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html) Disallow: / User-agent: GeonaBot 1.x; http://www.geona.com/ Disallow: / User-agent: Gigabot/2.0 (gigablast.com) Disallow: / User-agent: Gigabot/2.0/gigablast.com/spider.html Disallow: / User-agent: Gigabot/2.0; http://www.gigablast.com/spider.html Disallow: / User-agent: Gigabot/2.0att Disallow: / User-agent: Gigabot/3.0 (http://www.gigablast.com/spider.html) Disallow: / User-agent: Gigabot/x.0 Disallow: / User-agent: GigabotSiteSearch/2.0 (sitesearch.gigablast.com) Disallow: / User-agent: GoForIt.com Disallow: / User-agent: Goblin/0.9 (http://www.goguides.org/) Disallow: / User-agent: Goblin/0.9.x (http://www.goguides.org/goblin-info.html) Disallow: / User-agent: Goofer/0.2 Disallow: / User-agent: Googlebot-Image/1.0 Disallow: / User-agent: Googlebot-Image/1.0 ( http://www.googlebot.com/bot.html) Disallow: / User-agent: Googlebot/2.1 ( http://www.google.com/bot.html) Disallow: / User-agent: Googlebot/2.1 ( http://www.googlebot.com/bot.html) Disallow: / User-agent: Googlebot/Test ( http://www.googlebot.com/bot.html) Disallow: / User-agent: GrapeFX/0.3 libwww/5.4.0 Disallow: / User-agent: GrigorBot 0.8 (http://www.grigor.biz/bot.html) Disallow: / User-agent: Gromit/1.0 Disallow: / User-agent: Guestbook Auto Submitter Disallow: / User-agent: Gulliver/1.3 Disallow: / User-agent: Gulper Web Bot 0.2.4 (www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/Link/GulperBot) Disallow: / User-agent: Gungho/0.08004 (http://code.google.com/p/gungho-crawler/wiki/Index) Disallow: / User-agent: GurujiBot/1.0 (+http://www.guruji.com/WebmasterFAQ.html) Disallow: / User-agent: GurujiImageBot/1.0 (+http://www.guruji.com/en/WebmasterFAQ.html) Disallow: / User-agent: HPL/Nutch-0.9 - Disallow: / User-agent: HappyFunBot/1.1 Disallow: / User-agent: Harvest-NG/1.0.2 Disallow: / User-agent: Haste/0.12 (HOME: http://haste.kytoon.com/) Disallow: / User-agent: Hatena Antenna/0.4 (http://a.hatena.ne.jp/help#robot) Disallow: / User-agent: Hatena Pagetitle Agent/1.0 Disallow: / User-agent: Hatena RSS/0.3 (http://r.hatena.ne.jp) Disallow: / User-agent: HatenaScreenshot/1.0 (checker) Disallow: / User-agent: HeinrichderMiragoRobot Disallow: / User-agent: HeinrichderMiragoRobot (http://www.miragorobot.com/scripts/deinfo.asp) Disallow: / User-agent: Helix/1.x ( http://www.sitesearch.ca/helix/) Disallow: / User-agent: HenriLeRobotMirago (http://www.miragorobot.com/scripts/frinfo.asp) Disallow: / User-agent: HenryTheMiragoRobot (http://www.miragorobot.com/scripts/mrinfo.asp) Disallow: / User-agent: HenrytheMiragoRobot Disallow: / User-agent: Hi! I'm CsCrawler my homepage: http://www.kde.cs.uni-kassel.de/lehre/ss2005/googlespam/crawler.html RPT-HTTPClient/0.3-3 Disallow: / User-agent: Hippias/0.9 Beta Disallow: / User-agent: HitList Disallow: / User-agent: Hitwise Spider v1.0 http://www.hitwise.com Disallow: / User-agent: HomePageSearch(hpsearch.uni-trier.de) Disallow: / User-agent: Homerbot: www.homerweb.com Disallow: / User-agent: Honda-Search/0.7.2 (Nutch; http://lucene.apache.org/nutch/bot.html; search@honda-search.com) Disallow: / User-agent: HooWWWer/2.1.3 (debugging run) (+http://cosco.hiit.fi/search/hoowwwer/ | mailto:crawler-infohiit.fi) Disallow: / User-agent: HooWWWer/2.1.x ( http://cosco.hiit.fi/search/hoowwwer/ | mailto:crawler-infohiit.fi) Disallow: / User-agent: IAArchiver-1.0 Disallow: / User-agent: ICC-Crawler(Mozilla-compatible; http://kc.nict.go.jp/icc/crawl.html; icc-crawl(at)ml(dot)nict(dot)go(dot)jp) Disallow: / User-agent: ICC-Crawler(Mozilla-compatible;http://kc.nict.go.jp/icc/crawl.html;icc-crawl-contact(at)ml(dot)nict(dot)go(dot)jp) Disallow: / User-agent: ICCrawler - ICjobs (http://www.icjobs.de/bot.htm) Disallow: / User-agent: ICRA_label_spider/x.0 Disallow: / User-agent: IIITBOT/1.1 (Indian Language Web Search Engine; http://webkhoj.iiit.net; pvvpr at iiit dot ac dot in) Disallow: / User-agent: INFOMINE/8.0 Adders Disallow: / User-agent: INFOMINE/8.0 RemoteServices Disallow: / User-agent: INFOMINE/8.0 VLCrawler (http://infomine.ucr.edu/useragents) Disallow: / User-agent: INGRID/3.0 MT (webcrawler@NOSPAMexperimental.net; http://webmaster.ilse.nl/jsp/webmaster.jsp) Disallow: / User-agent: IP2LocationBot/1.0 http://www.ip2location.com" Disallow: / User-agent: IP2MapBot/1.1 http://www.ip2map.com Disallow: / User-agent: IPiumBot laurion(dot)com Disallow: / User-agent: IRLbot/1.0 ( http://irl.cs.tamu.edu/crawler) Disallow: / User-agent: IRLbot/3.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler/) Disallow: / User-agent: ISC Systems iRc Search 2.1 Disallow: / User-agent: IUPUI Research Bot v 1.9a Disallow: / User-agent: IWAgent/ 1.0 - www.brandprotect.com Disallow: / User-agent: IconSurf/2.0 favicon finder (see http://iconsurf.com/robot.html) Disallow: / User-agent: IconSurf/2.0 favicon monitor (see http://iconsurf.com/robot.html) Disallow: / User-agent: IlTrovatore-Setaccio ( http://www.iltrovatore.it) Disallow: / User-agent: IlTrovatore-Setaccio/1.2 ( http://www.iltrovatore.it/aiuto/faq.html) Disallow: / User-agent: IlTrovatore/1.2 (IlTrovatore; http://www.iltrovatore.it/bot.html; bot@iltrovatore.it) Disallow: / User-agent: IlseBot/1.x Disallow: / User-agent: Iltrovatore-Setaccio/0.3-dev (Indexing; http://www.iltrovatore.it/bot.html; info@iltrovatore.it) Disallow: / User-agent: Iltrovatore-Setaccio/1.2 (It-bot; http://www.iltrovatore.it/bot.html; info@iltrovatore.it) Disallow: / User-agent: ImageWalker/2.0 (www.bdbrandprotect.com) Disallow: / User-agent: IncyWincy data gatherer(webmaster@loopimprovements.com Disallow: / User-agent: IncyWincy page crawler(webmaster@loopimprovements.com Disallow: / User-agent: IncyWincy(http://www.look.com) Disallow: / User-agent: IncyWincy(http://www.loopimprovements.com/robot.html) Disallow: / User-agent: IncyWincy/2.1(loopimprovements.com/robot.html) Disallow: / User-agent: IndexTheWeb.com Crawler7 Disallow: / User-agent: Industry Program 1.0.x Disallow: / User-agent: Inet library Disallow: / User-agent: InfoFly/1.0 (http://www.versions-project.org/) Disallow: / User-agent: InfoNaviRobot(F107) Disallow: / User-agent: InfoSeek Sidewinder/0.9 Disallow: / User-agent: InfoSeek Sidewinder/1.0A Disallow: / User-agent: InfoSeek Sidewinder/1.1A Disallow: / User-agent: Infoseek SideWinder/1.45 (Compatible; MSIE 10.0; UNIX) Disallow: / User-agent: Infoseek SideWinder/2.0B (Linux 2.4 i686) Disallow: / User-agent: Inktomi Search Disallow: / User-agent: InnerpriseBot/1.0 (http://www.innerprise.com/) Disallow: / User-agent: Insitor.com search and find world wide! Disallow: / User-agent: Insitornaut Disallow: / User-agent: Internet Ninja x.0 Disallow: / User-agent: InternetArchive/0.8-dev(Nutch;http://lucene.apache.org/nutch/bot.html;nutch-agent@lucene.apache Disallow: / User-agent: InternetSeer.com Disallow: / User-agent: IpselonBot/0.xx-beta (Ipselon; http://www.ipselon.com; ipselonbot@ipselon.com) Disallow: / User-agent: Jabot/6.x (http://odin.ingrid.org/) Disallow: / User-agent: Jabot/7.x.x (http://odin.ingrid.org/) Disallow: / User-agent: Jack Disallow: / User-agent: Jambot/0.1.x (Jambot; http://www.jambot.com/blog; crawler@jambot.com) Disallow: / User-agent: Jambot/0.2.1 (Jambot; http://www.jambot.com/blog/static.php?page=webmaster-robot; crawler@jambot.com) Disallow: / User-agent: Java/1.4.1_01 Disallow: / User-agent: Java1.4.0 Disallow: / User-agent: Jayde Crawler. http://www.jayde.com Disallow: / User-agent: Jetbot/1.0 Disallow: / User-agent: JobSpider_BA/1.1 Disallow: / User-agent: Jyxobot/x Disallow: / User-agent: KAIST AITrc Crawler Disallow: / User-agent: KE_1.0/2.0 libwww/5.2.8 Disallow: / User-agent: KFSW-Bot (Version: 1.01 powered by KFSW www.kfsw.de) Disallow: / User-agent: KIT-Fireball/2.0 Disallow: / User-agent: KIT-Fireball/2.0 (compatible; Mozilla 4.0; MSIE 5.5) Disallow: / User-agent: KSbot/1.0 (KnowledgeStorm crawler; http://www.knowledgestorm.com/resources/content/crawler/index.html; crawleradmin@knowledgestorm.com) Disallow: / User-agent: KakleBot - www.kakle.com/0.1 (KakleBot - www.kakle.com; http:// www.kakle.com/bot.html; support@kakle.com) Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Kevin http://dznet.com/kevin/ Disallow: / User-agent: Kevin http://websitealert.net/kevin/ Disallow: / User-agent: KnowItAll(knowitall@cs.washington.edu) Disallow: / User-agent: Knowledge.com/0.x Disallow: / User-agent: Krugle/Krugle,Nutch/0.8+ (Krugle web crawler; http://www.krugle.com/crawler/info.html; webcrawler@krugle.com) Disallow: / User-agent: LARBIN-EXPERIMENTAL (efp@gmx.net) Disallow: / User-agent: LECodeChecker/3.0 libgetdoc/1.0 Disallow: / User-agent: LEIA/2.90 Disallow: / User-agent: LEIA/3.01pr (LEIAcrawler; [SNIP]) Disallow: / User-agent: LMQueueBot/0.2 Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: LTI/LemurProject Nutch Spider/Nutch-1.0-dev (Research spider using Nutch; http://www.lemurproject.org; mhoy@cs.cmu.edu) Disallow: / User-agent: LTI/LemurProject Nutch Spider/Nutch-1.0-dev (lti crawler for CMU; http://www.lti.cs.cmu.edu; changkuk at cmu dot edu) Disallow: / User-agent: LWP::Simple/5.22 Disallow: / User-agent: LWP::Simple/5.36 Disallow: / User-agent: LWP::Simple/5.48 Disallow: / User-agent: LWP::Simple/5.50 Disallow: / User-agent: LWP::Simple/5.51 Disallow: / User-agent: LWP::Simple/5.53 Disallow: / User-agent: LWP::Simple/5.63 Disallow: / User-agent: LWP::Simple/5.803 Disallow: / User-agent: LapozzBot/1.4 ( http://robot.lapozz.com) Disallow: / User-agent: LapozzBot/1.5 (+http://robot.lapozz.hu) Disallow: / User-agent: LetsCrawl.com/1.0 +http://letscrawl.com/ Disallow: / User-agent: LexiBot/1.00 Disallow: / User-agent: Libby_1.1/libwww-perl/5.47 Disallow: / User-agent: LibertyW (+http://www.lw01.com) Disallow: / User-agent: LijitSpider/Nutch-0.9 (Reports crawler; http://www.lijit.com/; info(a)lijit(d)com) Disallow: / User-agent: Lincoln State Web Browser Disallow: / User-agent: LinkScan/11.0beta2 UnixShareware robot from Elsop.com (used by Indiafocus/Indiainfo) Disallow: / User-agent: LinkScan/9.0g Unix Disallow: / User-agent: LinkScan/x.x Unix Disallow: / User-agent: Links 2.0 (http://gossamer-threads.com/scripts/links/) Disallow: / User-agent: Links SQL (http://gossamer-threads.com/scripts/links-sql/) Disallow: / User-agent: ListBidBot (freelance job spider http://listbid.com)Freelance Disallow: / User-agent: LiveTrans/Nutch-0.9 (maintainer: cobain at iis dot sinica dot edu dot tw; http://wkd.iis.sinica.edu.tw/LiveTrans/) Disallow: / User-agent: Llaut/1.0 (http://mnm.uib.es/~gallir/llaut/bot.html) Disallow: / User-agent: LocalBot/1.0 ( http://www.localbot.co.uk/) Disallow: / User-agent: LocalcomBot/1.2.x ( http://www.local.com/bot.htm) Disallow: / User-agent: Lockstep Spider/1.0 Disallow: / User-agent: Look.com Disallow: / User-agent: Lovel as 1.0 ( +http://www.everatom.com) Disallow: / User-agent: Lycos_Spider_(T-Rex) Disallow: / User-agent: Lycos_Spider_(modspider) Disallow: / User-agent: Lynx/2.8.4rel.1 libwww-FM/2.14 SSL-MM/1.4.1 OpenSSL/0.9.6c (human-guided@lerly.net) Disallow: / User-agent: MARTINI Disallow: / User-agent: MDbot/1.0 (+http://www.megadownload.net/bot.html) Disallow: / User-agent: MFC Foundation Class Library 4.0 Disallow: / User-agent: MJ12bot/vx.x.x (http://majestic12.co.uk/bot.php?+) Disallow: / User-agent: MJ12bot/vx.x.x (http://www.majestic12.co.uk/projects/dsearch/mj12bot.php) Disallow: / User-agent: MJBot (SEO assessment) Disallow: / User-agent: MLBot (www.metadatalabs.com) Disallow: / User-agent: MQBOT/Nutch-0.9-dev (MQBOT Nutch Crawler; http://falcon.cs.uiuc.edu; mqbot@cs.uiuc.edu) Disallow: / User-agent: MQbot metaquerier.cs.uiuc.edu/crawler Disallow: / User-agent: MSNBOT/0.xx (http://search.msn.com/msnbot.htm) Disallow: / User-agent: MSNBOT_Mobile MSMOBOT Mozilla/2.0 (compatible; MSIE 4.02; Windows CE; Default) Disallow: / User-agent: MSNPTC/1.0 Disallow: / User-agent: MSRBOT (http://research.microsoft.com/research/sv/msrbot) Disallow: / User-agent: MVAClient Disallow: / User-agent: MaSagool/1.0 (MaSagool; http://sagool.jp/; info@sagool.jp) Disallow: / User-agent: Mac Finder 1.0.xx Disallow: / User-agent: Mackster( http://www.ukwizz.com ) Disallow: / User-agent: Mahiti.Com/Mahiti Crawler-1.0 (Mahiti.Com; http://mahiti.com ; mahiti.com) Disallow: / User-agent: Mail.Ru/1.0 Disallow: / User-agent: MantraAgent Disallow: / User-agent: MapoftheInternet.com ( http://MapoftheInternet.com) Disallow: / User-agent: Mariner/5.1b [de] (Win95; I ;Kolibri gncwebbot) Disallow: / User-agent: Marketwave Hit List Disallow: / User-agent: Martini Disallow: / User-agent: Marvin v0.3 Disallow: / User-agent: MasterSeek Disallow: / User-agent: Mata Hari/2.00 Disallow: / User-agent: Matrix S.p.A. - FAST Enterprise Crawler 6 (Unknown admin e-mail address) Disallow: / User-agent: MediaCrawler-1.0 (Experimental) Disallow: / User-agent: MediaSearch/0.1 Disallow: / User-agent: Mediapartners-Google/2.1 ( http://www.googlebot.com/bot.html) Disallow: / User-agent: MegaSheep v1.0 (www.searchuk.com internet sheep) Disallow: / User-agent: Megite2.0 (http://www.megite.com) Disallow: / User-agent: Mercator-1.x Disallow: / User-agent: Mercator-2.0 Disallow: / User-agent: Mercator-Scrub-1.1 Disallow: / User-agent: MetaGer-LinkChecker Disallow: / User-agent: MetaGer_PreChecker0.1 Disallow: / User-agent: Metaeuro Web Crawler/0.2 (MetaEuro Web Search Clustering Engine; http://www.metaeuro.com; crawler at metaeuro dot com) Disallow: / User-agent: MetagerBot/0.8-dev (MetagerBot; http://metager.de; ) Disallow: / User-agent: Metaspinner/0.01 (Metaspinner; http://www.meta-spinner.de/; support@meta-spinner.de/) Disallow: / User-agent: MicroBaz Disallow: / User-agent: Microsoft Small Business Indexer Disallow: / User-agent: Microsoft URL Control - 6.00.8xxx Disallow: / User-agent: MicrosoftPrototypeCrawler (How's my crawling? mailto:newbiecrawler@hotmail.com) Disallow: / User-agent: Missauga Locate 1.0.0 Disallow: / User-agent: Missigua Locator 1.9 Disallow: / User-agent: Missouri College Browse Disallow: / User-agent: Misterbot-Nutch/0.7.1 (Misterbot-Nutch; http://www.misterbot.fr; admin@misterbot.fr) Disallow: / User-agent: Miva (AlgoFeedback@miva.com) Disallow: / User-agent: Mizzu Labs 2.2 Disallow: / User-agent: MnogoSearch/3.2.xx Disallow: / User-agent: Mo College 1.9 Disallow: / User-agent: MojeekBot/0.x (archi; http://www.mojeek.com/bot.html) Disallow: / User-agent: Morris - Mixcat Crawler ( http://mixcat.com) Disallow: / User-agent: Mouse-House/7.4 (spider_monkey spider info at www.mobrien.com/sm.shtml) Disallow: / User-agent: Mozilla Disallow: / User-agent: MultiText/0.1 Disallow: / User-agent: MusicWalker2.0 ( http://www.somusical.com) Disallow: / User-agent: Mylinea.com Crawler 2.0 Disallow: / User-agent: NABOT/5.0 Disallow: / User-agent: NASA Search 1.0 Disallow: / User-agent: NCSA Beta 1 (http://vias.ncsa.uiuc.edu/viasarchivinginformation.html) Disallow: / User-agent: NEC Research Agent -- compuman at research.nj.nec.com Disallow: / User-agent: NG-Search/0.90 (NG-SearchBot; http://www.ng-search.com; ) Disallow: / User-agent: NG/1.0 Disallow: / User-agent: NG/4.0.1229 Disallow: / User-agent: NICO/1.0 Disallow: / User-agent: NITLE Blog Spider/0.01 Disallow: / User-agent: NP/0.1 (NP; http://www.nameprotect.com; npbot@nameprotect.com) Disallow: / User-agent: NPBot (http://www.nameprotect.com/botinfo.html) Disallow: / User-agent: NPBot-1/2.0 Disallow: / User-agent: Naamah 1.0a/Blogbot (http://blogbot.de/) Disallow: / User-agent: NameOfAgent (CMS Spider) Disallow: / User-agent: NationalDirectory-WebSpider/1.3 Disallow: / User-agent: NationalDirectoryAddURL/1.0 Disallow: / User-agent: NaverBot-1.0 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com) Disallow: / User-agent: NaverBot_dloader/1.5 Disallow: / User-agent: NavissoBot Disallow: / User-agent: NavissoBot/1.7 (+http://navisso.com/) Disallow: / User-agent: Nebullabot/2.2 (http://bot.nebulla.info) Disallow: / User-agent: NetLookout/2.24 Disallow: / User-agent: NetNoseCrawler/v1.0 Disallow: / User-agent: NetResearchServer(http://www.look.com) Disallow: / User-agent: NetResearchServer/x.x(loopimprovements.com/robot.html) Disallow: / User-agent: NetSprint -- 2.0 Disallow: / User-agent: NetWhatCrawler/0.06-dev (NetWhatCrawler from NetWhat.com; http://www.netwhat.com; support@netwhat.com) Disallow: / User-agent: NetZippy Disallow: / User-agent: NetinfoBot/1.0 (http://netinfo.bg/netinfobot.html) Disallow: / User-agent: Netluchs/0.8-dev ( ; http://www.netluchs.de/; ___don't___spam_me_@netluchs.de) Disallow: / User-agent: Netprospector JavaCrawler Disallow: / User-agent: NextGenSearchBot 1 (for information visit http://www.eliyon.com/NextGenSearchBot) Disallow: / User-agent: NextopiaBOT (+http://www.nextopia.com) distributed crawler client beta v0.x Disallow: / User-agent: Noago Spider Disallow: / User-agent: Nokia-WAPToolkit/1.2 googlebot(at)googlebot.com Disallow: / User-agent: Nokia6610/1.0 (3.09) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/crawling/crawling-01.html) Disallow: / User-agent: NokodoBot/1.x (+http://nokodo.com/bot.htm) Disallow: / User-agent: Norbert the Spider(Burf.com) Disallow: / User-agent: Nsauditor/1.x Disallow: / User-agent: NuSearch Spider (compatible; MSIE 6.0) Disallow: / User-agent: NuSearch Spider www.nusearch.com Disallow: / User-agent: Nutch Disallow: / User-agent: Nutch crawler/Nutch-0.9 (picapage.com; admin@picapage.com) Disallow: / User-agent: Nutch/Nutch-0.9 (Eurobot; http://www.ayell.eu ) Disallow: / User-agent: NutchCVS/0.0x-dev (Nutch; http://www.nutch.org/docs/bot.html; nutch-agent@lists.sourceforge.net) Disallow: / User-agent: NutchCVS/0.7.1 (Nutch running at UW; http://www.nutch.org/docs/en/bot.html; sycrawl@cs.washington.edu) Disallow: / User-agent: NutchEC2Test/Nutch-0.9-dev (Testing Nutch on Amazon EC2.; http://lucene.apache.org/nutch/bot.html; ec2test at lucene.com) Disallow: / User-agent: NutchOrg/0.0x-dev (Nutch; http://www.nutch.org/docs/bot.html; nutch-agent@lists.sourceforge.net) Disallow: / User-agent: NutchVinegarCrawl/Nutch-0.8.1 (Vinegar; http://www.cs.washington.edu; eytanadar at gmail dot com) Disallow: / User-agent: ObjectsSearch/0.01-dev (ObjectsSearch;http://www.ObjectsSearch.com/bot.html; support@thesoftwareobjects.com) Disallow: / User-agent: ObjectsSearch/0.0x (ObjectsSearch; http://www.ObjectsSearch.com/bot.html; support@thesoftwareobjects.com) Disallow: / User-agent: Ocelli/1.x (http://www.globalspec.com/Ocelli) Disallow: / User-agent: Octora Beta - www.octora.com Disallow: / User-agent: Octora Beta Bot - www.octora.com Disallow: / User-agent: OliverPerry Disallow: / User-agent: OmniExplorer_Bot/1.0x (+http://www.omni-explorer.com) Internet CategorizerOmniExplorer http://www.omni-explorer.com/ car & shopping search (64.62.175.xxx) Disallow: / User-agent: OmniExplorer_Bot/1.0x (+http://www.omni-explorer.com) Job Crawler Disallow: / User-agent: OmniExplorer_Bot/1.1x (+http://www.omni-explorer.com) Torrent Crawler Disallow: / User-agent: OmniExplorer_Bot/x.xx (+http://www.omni-explorer.com) WorldIndexer Disallow: / User-agent: Onet.pl SA- http://szukaj.onet.pl Disallow: / User-agent: OntoSpider/1.0 libwww-perl/5.65 Disallow: / User-agent: OpenAcoon v4.0.x (www.openacoon.de) Disallow: / User-agent: OpenISearch/1.x (www.openisearch.com) Disallow: / User-agent: OpenTaggerBot (http://www.opentagger.com/opentaggerbot.htm) Disallow: / User-agent: OpenTextSiteCrawler/2.9.2 Disallow: / User-agent: OpenWebSpider/0.x.x (http://www.openwebspider.org) Disallow: / User-agent: OpenWebSpider/x Disallow: / User-agent: Openbot/3.0+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/robot.html) Disallow: / User-agent: Openfind Robot/1.1A2 Disallow: / User-agent: Openfind data gatherer- Openbot/3.0+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/robot.html) Disallow: / User-agent: OpidooBOT (larbin2.6.3@unspecified.mail) Disallow: / User-agent: Oracle Ultra Search Disallow: / User-agent: OrangeSpider Disallow: / User-agent: Orbiter/T-2.0 (+http://www.dailyorbit.com/bot.htm) Disallow: / User-agent: Overture-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; http://fast.no/support/crawler.asp) Disallow: / User-agent: PADLibrary Spider Disallow: / User-agent: PBrowse 1.4b Disallow: / User-agent: PEERbot www.peerbot.com Disallow: / User-agent: PEval 1.4b Disallow: / User-agent: PJspider/3.0 (pjspider@portaljuice.com; http://www.portaljuice.com) Disallow: / User-agent: PRCrawler/Nutch-0.9 (data mining development project; crawler@projectrialto.com) Disallow: / User-agent: PROve AnswerBot 4.0 Disallow: / User-agent: PSurf15a 11 Disallow: / User-agent: PSurf15a 51 Disallow: / User-agent: PSurf15a VA Disallow: / User-agent: PWeBot/1.2 Inspector (http://www.programacionweb.net/robot.php) Disallow: / User-agent: PageBitesHyperBot/600 (http://www.pagebites.com/) Disallow: / User-agent: Pagebull http://www.pagebull.com/ Disallow: / User-agent: ParaSite/1.0b (http://www.ianett.com/parasite/) Disallow: / User-agent: Patwebbot (http://www.herz-power.de/technik.html) Disallow: / User-agent: PicoSearch/1.0 Disallow: / User-agent: Piffany_Web_Scraper_v0.x Disallow: / User-agent: Piffany_Web_Spider_v0.x Disallow: / User-agent: PigeonBot1.0 BETA Disallow: / User-agent: PingALink Monitoring Services 1.0 Disallow: / User-agent: PingALink Monitoring Services 1.0 (http://www.pingalink.com) Disallow: / User-agent: Pita Disallow: / User-agent: PlagiarBot/1.0 Disallow: / User-agent: PlantyNet_WebRobot_V1.9 dhkang@plantynet.com Disallow: / User-agent: PluckFeedCrawler/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://www.pluck.com; 1 subscribers) Disallow: / User-agent: Pluggd/Nutch-0.9 (automated crawler http://www.pluggd.com;support at pluggd dot com) Disallow: / User-agent: Poirot Disallow: / User-agent: Pompos/1.x http://dir.com/pompos.html Disallow: / User-agent: Pompos/1.x pompos@iliad.fr Disallow: / User-agent: Popdexter/1.0 Disallow: / User-agent: Port Huron Labs Disallow: / User-agent: PortalBSpider/2.0 (spider@portalb.com) Disallow: / User-agent: PrivacyFinder Cache Bot v1.0 Disallow: / User-agent: PrivacyFinder/1.1 Disallow: / User-agent: ProWebGuide Link Checker (http://www.prowebguide.com) Disallow: / User-agent: Production Bot 0116B Disallow: / User-agent: Production Bot 2016B Disallow: / User-agent: Production Bot DOT 3016B Disallow: / User-agent: Program Shareware 1.0.2 Disallow: / User-agent: Project XP5 [2.03.07-111203] Disallow: / User-agent: PubCrawl (pubcrawl.stanford.edu) Disallow: / User-agent: PycURL Disallow: / User-agent: Python-urllib/1.1x Disallow: / User-agent: Python-urllib/2.0a1 Disallow: / User-agent: QEAVis Agent/Nutch-0.9 (Quantitative Evaluation of Academic Websites Visibility; http://nlp.uned.es/qeavis Disallow: / User-agent: QPCreep Test Rig ( We are not indexing- just testing ) Disallow: / User-agent: Qango.com Web Directory (http://www.qango.com/) Disallow: / User-agent: QuepasaCreep ( crawler@quepasacorp.com ) Disallow: / User-agent: QuepasaCreep v0.9.1x Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: QweeryBot/3.01 ( http://qweerybot.qweery.nl) Disallow: / User-agent: Qweery_robot.txt_CheckBot/3.01 (http://qweerybot.qweery.com) Disallow: / User-agent: R6_CommentReader_(www.radian6.com/crawler) Disallow: / User-agent: R6_FeedFetcher_(www.radian6.com/crawler) Disallow: / User-agent: RAMPyBot - www.giveRAMP.com/0.1 (RAMPyBot - www.giveRAMP.com; http://www.giveramp.com/bot.html; support@giveRAMP.com) Disallow: / User-agent: RAMPyBot/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org) Disallow: / User-agent: RRC (crawler_admin@bigfoot.com) Disallow: / User-agent: RSSMicro.com RSS/Atom Feed Robot Disallow: / User-agent: RSurf15a 41 Disallow: / User-agent: RSurf15a 51 Disallow: / User-agent: RSurf15a 81 Disallow: / User-agent: RaBot/1.0 Agent-admin/phortse@hanmail.net Disallow: / User-agent: Rankivabot/3.2 (www.rankiva.com; 3.2; vzmxikn) Disallow: / User-agent: Rational SiteCheck (Windows NT) Disallow: / User-agent: Reaper [2.03.10-031204] (http://www.sitesearch.ca/reaper/) Disallow: / User-agent: Reaper/2.0x (+http://www.sitesearch.ca/reaper) Disallow: / User-agent: RebusnetBot (+http://www.rebusnet.biz) Disallow: / User-agent: RebusnetPADBot/1.5x (+http://www.rebusnet.biz) Disallow: / User-agent: RedCarpet/1.2 (http://www.redcarpet-inc.com/robots.html) Disallow: / User-agent: RedCell/0.1 (InfoSec Search Bot (Coming Soon); http://www.telegenetic.net/bot.html; lhall@telegenetic.net) Disallow: / User-agent: RedCell/0.1 (RedCell; telegenetic.net/bot.html; lhall_at_telegenetic.net) Disallow: / User-agent: RedKernel WWW-Spider 2/0 (+http://www-spider.redkernel-softwares.com/) Disallow: / User-agent: RixBot (http://babelserver.org/rix) Disallow: / User-agent: RoboCrawl (http://www.canadiancontent.net) Disallow: / User-agent: RoboCrawl (www.canadiancontent.net) Disallow: / User-agent: RoboPal (http://www.findpal.com/) Disallow: / User-agent: Robot/www.pj-search.com Disallow: / User-agent: Robot: NutchCrawler- Owner: wdavies@acm.org Disallow: / User-agent: Robot@SuperSnooper.Com Disallow: / User-agent: Robozilla/1.0 Disallow: / User-agent: Rotondo/3.1 libwww/5.3.1 Disallow: / User-agent: RufusBot (Rufus Web Miner; http://64.124.122.252/feedback.html) Disallow: / User-agent: RufusBot (Rufus Web Miner; http://www.webaroo.com/rooSiteOwners.html) Disallow: / User-agent: SBIder/0.7 (SBIder; http://www.sitesell.com/sbider.html; http://support.sitesell.com/contact-support.html) Disallow: / User-agent: SBIder/0.8-dev (SBIder; http://www.sitesell.com/sbider.html; http://support.sitesell.com/contact-support.html) Disallow: / User-agent: SSurf15a 11 Disallow: / User-agent: SWSBot-Images/1.2 http://www.smartwaresoft.com/swsbot12.html Disallow: / User-agent: SandCrawler - Compatibility Testing Disallow: / User-agent: ScanWeb Disallow: / User-agent: ScholarUniverse/0.8 (Nutch;+http://scholaruniverse.com/bot.jsp; fetch-agent@scholaruniverse.com) Disallow: / User-agent: ScollSpider/2.0 (+http://www.webwobot.com/ScollSpider.php) Disallow: / User-agent: Scooter-3.0.EU Disallow: / User-agent: Scooter-3.0.FS Disallow: / User-agent: Scooter-3.0.HD Disallow: / User-agent: Scooter-3.0.VNS Disallow: / User-agent: Scooter-3.0QI Disallow: / User-agent: Scooter-3.2 Disallow: / User-agent: Scooter-3.2.BT Disallow: / User-agent: Scooter-3.2.DIL Disallow: / User-agent: Scooter-3.2.EX Disallow: / User-agent: Scooter-3.2.JT Disallow: / User-agent: Scooter-3.2.NIV Disallow: / User-agent: Scooter-3.2.SF0 Disallow: / User-agent: Scooter-3.2.snippet Disallow: / User-agent: Scooter-3.3dev Disallow: / User-agent: Scooter-ARS-1.1 Disallow: / User-agent: Scooter-ARS-1.1-ih Disallow: / User-agent: Scooter-W3-1.0 Disallow: / User-agent: Scooter-W3.1.2 Disallow: / User-agent: Scooter/1.0 Disallow: / User-agent: Scooter/1.0 scooter@pa.dec.com Disallow: / User-agent: Scooter/1.1 (custom) Disallow: / User-agent: Scooter/2.0 G.R.A.B. V1.1.0 Disallow: / User-agent: Scooter/2.0 G.R.A.B. X2.0 Disallow: / User-agent: Scooter/3.3 Disallow: / User-agent: Scooter/3.3.QA.pczukor Disallow: / User-agent: Scooter/3.3.vscooter Disallow: / User-agent: Scooter/3.3_SF Disallow: / User-agent: Scooter2_Mercator_x-x.0 Disallow: / User-agent: Scooter_bh0-3.0.3 Disallow: / User-agent: Scooter_trk3-3.0.3 Disallow: / User-agent: ScoutAbout Disallow: / User-agent: ScoutAnt/0.1; +http://www.ant.com/what_is_ant.com/ Disallow: / User-agent: Scrubby/2.x (http://www.scrubtheweb.com/) Disallow: / User-agent: Scrubby/3.0 (+http://www.scrubtheweb.com/help/technology.html) Disallow: / User-agent: Search+ Disallow: / User-agent: Search-Engine-Studio Disallow: / User-agent: Search/1.0 (http://www.innerprise.net/es-spider.asp) Disallow: / User-agent: SearchByUsa/2 (SearchByUsa; http://www.SearchByUsa.com/bot.html; info@SearchByUsa.com) Disallow: / User-agent: SearchExpress Spider0.99 Disallow: / User-agent: SearchGuild/DMOZ/Experiment (searchguild@gmail.com) Disallow: / User-agent: SearchGuild_DMOZ_Experiment (chris@searchguild.com) Disallow: / User-agent: SearchSight/2.0 (http://SearchSight.com/) Disallow: / User-agent: SearchSpider.com/1.1 Disallow: / User-agent: SearchTone2.0 - IDEARE Disallow: / User-agent: SearchdayBot Disallow: / User-agent: Searchit-Now Robot/2.2 (+http://www.searchit-now.co.uk) Disallow: / User-agent: Searchmee! Spider v0.98a Disallow: / User-agent: Searchspider/1.2 (SearchSpider; http://www.searchspider.com; webmaster@searchspider.com) Disallow: / User-agent: Seekbot/1.0 (http://www.seekbot.net/bot.html) HTTPFetcher/0.3 Disallow: / User-agent: Seekbot/1.0 (http://www.seekbot.net/bot.html) RobotsTxtFetcher/1.0 (XDF) Disallow: / User-agent: Seekbot/1.0 (http://www.seekbot.net/bot.html) RobotsTxtFetcher/1.2 Disallow: / User-agent: Seeker.lookseek.com Disallow: / User-agent: Semager/1.1 (http://www.semager.de/blog/semager-bots/) Disallow: / User-agent: Semager/1.x (http://www.semager.de) Disallow: / User-agent: Sensis Web Crawler (search_comments\at\sensis\dot\com\dot\au) Disallow: / User-agent: Sensis.com.au Web Crawler (search_comments\at\sensis\dot\com\dot\au) Disallow: / User-agent: SeznamBot/1.0 Disallow: / User-agent: SeznamBot/1.0 (+http://fulltext.seznam.cz/) Disallow: / User-agent: SeznamBot/2.0-test (+http://fulltext.sblog.cz/) Disallow: / User-agent: ShablastBot 1.0 Disallow: / User-agent: SharewarePlazaFileCheckBot/1.0+(+http://www.SharewarePlaza.com) Disallow: / User-agent: Shim Crawler Disallow: / User-agent: Shim-Crawler(Mozilla-compatible; http://www.logos.ic.i.u-tokyo.ac.jp/crawler/; crawl@logos.ic.i.u-tokyo.ac.jp) Disallow: / User-agent: ShopWiki/1.0 ( +http://www.shopwiki.com/) Disallow: / User-agent: ShopWiki/1.0 ( +http://www.shopwiki.com/wiki/Help:Bot) Disallow: / User-agent: Shoula.com Crawler 2.0 Disallow: / User-agent: SietsCrawler/1.1 (+http://www.siets.biz) Disallow: / User-agent: Sigram/Nutch-1.0-dev (Test agent for Nutch development; http://www.sigram.com/bot.html; bot at sigram dot com) Disallow: / User-agent: Siigle Orumcex v.001 Turkey (http://www.siigle.com) Disallow: / User-agent: Simpy 1.x; http://www.simpy.com/ Disallow: / User-agent: Simpy/1.x (Simpy; http://www.simpy.com/?ref=bot; feedback at simpy dot com) Disallow: / User-agent: Sirketcebot/v.01 (http://www.sirketce.com/bot.html) Disallow: / User-agent: SiteRecon+(xx) Disallow: / User-agent: SiteSpider +(http://www.SiteSpider.com/) Disallow: / User-agent: SiteTruth.com site rating system Disallow: / User-agent: SiteXpert Disallow: / User-agent: Skampy/0.9.x (http://www.skaffe.com/skampy-info.html) Disallow: / User-agent: Skimpy/0.x (http://www.skaffe.com/skampy-info.html) Disallow: / User-agent: Skywalker/0.1 (Skywalker; anonymous; anonymous) Disallow: / User-agent: Slarp/0.1 Disallow: / User-agent: Slider_Search_v1-de Disallow: / User-agent: Slurp/2.0 (slurp@inktomi.com; http://www.inktomi.com/slurp.html) Disallow: / User-agent: Slurp/2.0-KiteWeekly (slurp@inktomi.com; http://www.inktomi.com/slurp.html) Disallow: / User-agent: Slurp/si (slurp@inktomi.com; http://www.inktomi.com/slurp.html) Disallow: / User-agent: Slurpy Verifier/1.0 Disallow: / User-agent: SlySearch (slysearch@slysearch.com) Disallow: / User-agent: SlySearch/1.0 http://www.plagiarism.org/crawler/robotinfo.html Disallow: / User-agent: SlySearch/1.x http://www.slysearch.com Disallow: / User-agent: SmiffyDCMetaSpider/1.0 Disallow: / User-agent: Snapbot/1.0 Disallow: / User-agent: Snapbot/1.0 (Snap Shots, +http://www.snap.com) Disallow: / User-agent: SnykeBot/0.6 (http://www.snyke.com) Disallow: / User-agent: SocSciBot () Disallow: / User-agent: SoftHypermarketFileCheckBot/1.0+(+http://www.softhypermaket.com) Disallow: / User-agent: Sogou Orion spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07) Disallow: / User-agent: Sogou web spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07) Disallow: / User-agent: Sosospider+(+http://help.soso.com/webspider.htm) Disallow: / User-agent: Speedy Spider (Beta/x.x; speedy@entireweb.com) Disallow: / User-agent: Speedy Spider (Entireweb; Beta/1.0; http://www.entireweb.com/about/search_tech/speedyspider/) Disallow: / User-agent: Speedy_Spider (http://www.entireweb.com) Disallow: / User-agent: Sphere Scout&v4.0 - scout at sphere dot com Disallow: / User-agent: Sphider Disallow: / User-agent: Spida/0.1 Disallow: / User-agent: Spider-Sleek/2.0 (+http://search-info.com/linktous.html) Disallow: / User-agent: Spider/maxbot.com admin@maxbot.com Disallow: / User-agent: SpiderKU/0.x Disallow: / User-agent: SpiderMan Disallow: / User-agent: SpiderMonkey/7.0x (SpiderMonkey.ca info at http://spidermonkey.ca/sm.shtml) Disallow: / User-agent: Spinne/2.0 Disallow: / User-agent: Spinne/2.0 med Disallow: / User-agent: Spinne/2.0 med_AH Disallow: / User-agent: Spock Crawler (http://www.spock.com/crawler) Disallow: / User-agent: Sqworm/2.9.81-BETA (beta_release; 20011102-760; i686-pc-linux-gnu) Disallow: / User-agent: Sqworm/2.9.85-BETA (beta_release; 20011115-775; i686-pc-linux-gnu) Disallow: / User-agent: StackRambler/x.x Disallow: / User-agent: Steeler/1.x (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/) Disallow: / User-agent: Steeler/3.3 (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/) Disallow: / User-agent: Strategic Board Bot (+http://www.strategicboard.com) Disallow: / User-agent: Submission Spider at surfsafely.com Disallow: / User-agent: Suchknecht.at-Robot Disallow: / User-agent: SurferF3 1/0 Disallow: / User-agent: SurveyBot/2.2 Whois Source Disallow: / User-agent: SurveyBot/2.3 (Whois Source) Disallow: / User-agent: Swooglebot/2.0. (+http://swoogle.umbc.edu/swooglebot.htm) Disallow: / User-agent: SygolBot http://www.sygol.net Disallow: / User-agent: SynoBot Disallow: / User-agent: Syntryx ANT Scout Chassis Pheromone; Mozilla/4.0 compatible crawler Disallow: / User-agent: Szukacz/1.x Disallow: / User-agent: Szukacz/1.x (robot; www.szukacz.pl/jakdzialarobot.html; szukacz@proszynski.pl) Disallow: / User-agent: TAMU_CS_IRL_CRAWLER/1.0 Disallow: / User-agent: TCDBOT/Nutch-0.8 (PhD student research;"http://www.tcd.ie; mcgettrs at t c d dot IE)" Disallow: / User-agent: TECOMAC-Crawler/0.x Disallow: / User-agent: TJG/Spider Disallow: / User-agent: TSurf15a 11 Disallow: / User-agent: Tagword (http://tagword.com/dmoz_survey.php) Disallow: / User-agent: Talkro Web-Shot/1.0 (E-mail: webshot@daumsoft.com- Home: http://222.122.15.190/webshot) Disallow: / User-agent: Tecomi Bot (http://www.tecomi.com/bot.htm) Disallow: / User-agent: Teemer (NetSeer, Inc. is a Los Angeles based Internet startup company.; http://www.netseer.com/crawler.html; crawler@netseer.com) Disallow: / User-agent: Teoma MP Disallow: / User-agent: Teradex Mapper; mapper@teradex.com; http://www.teradex.com Disallow: / User-agent: TerrawizBot/1.0 (+http://www.terrawiz.com/bot.html) Disallow: / User-agent: Test spider Disallow: / User-agent: TestCrawler/Nutch-0.9 (Testing Crawler for Research ; http://balihoo.com/index.aspx; tgautier at balihoo dot com) Disallow: / User-agent: TheRarestParser/0.2a (http://therarestwords.com/) Disallow: / User-agent: TheSuBot/0.1 (www.thesubot.de) Disallow: / User-agent: TinEye/1.1 (http://tineye.com/crawler.html) Disallow: / User-agent: Tkensaku/x.x(http://www.tkensaku.com/q.html) Disallow: / User-agent: Topodia/1.2-dev (Topodia - Crawler for HTTP content indexing; http://www.topodia.com/; support@topodia.com) Disallow: / User-agent: Toutatis x-xx.x (hoppa.com) Disallow: / User-agent: Toutatis x.x (hoppa.com) Disallow: / User-agent: Toutatis x.x-x Disallow: / User-agent: Trampelpfad-Spider Disallow: / User-agent: Trampelpfad-Spider-v0.1 Disallow: / User-agent: Tumblr/1.0 RSS syndication (+http://www.tumblr.com/) (support@tumblr.com) Disallow: / User-agent: TurnitinBot/x.x (http://www.turnitin.com/robot/crawlerinfo.html) Disallow: / User-agent: Turnpike Emporium LinkChecker/0.1 Disallow: / User-agent: TutorGig/1.5 (+http://www.tutorgig.com/crawler) Disallow: / User-agent: Tutorial Crawler 1.4 (http://www.tutorgig.com/crawler) Disallow: / User-agent: Twiceler www.cuill.com/robots.html Disallow: / User-agent: Twiceler-0.9 http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: Tycoon Agent/Nutch-1.0-dev Disallow: / User-agent: TygoBot Disallow: / User-agent: TygoProwler Disallow: / User-agent: UIowaCrawler/1.0 Disallow: / User-agent: UKWizz/Nutch-0.8.1 (UKWizz Nutch crawler; http://www.ukwizz.com/) Disallow: / User-agent: URL Spider Pro/x.xx (innerprise.net) Disallow: / User-agent: URL_Spider_Pro/x.x Disallow: / User-agent: URL_Spider_Pro/x.x+(http://www.innerprise.net/usp-spider.asp) Disallow: / User-agent: USyd-NLP-Spider (http://www.it.usyd.edu.au/~vinci/bot.html) Disallow: / User-agent: UdmSearch/3.1.x Disallow: / User-agent: Ultraseek Disallow: / User-agent: Under the Rainbow 2.2 Disallow: / User-agent: UofTDB_experiment (leehyun@cs.toronto.edu) Disallow: / User-agent: UptimeBot(www.uptimebot.com) Disallow: / User-agent: Uptimebot Disallow: / User-agent: User-Agent: Mozilla/4.0 (SKIZZLE! Distributed Internet Spider v1.0 - www.SKIZZLE.com) Disallow: / User-agent: User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) Disallow: / User-agent: VMBot/0.x.x (VMBot; http://www.VerticalMatch.com/; vmbot@tradedot.com) Disallow: / User-agent: VSE/1.0 (testcrawler@hotmail.com) Disallow: / User-agent: VSE/1.0 (testcrawler@vivisimo.com) Disallow: / User-agent: VWBOT/Nutch-0.9-dev (VWBOT Nutch Crawler; http://vwbot.cs.uiuc.edu;+vwbot@cs.uiuc.edu Disallow: / User-agent: VadixBot Disallow: / User-agent: Vagabondo-WAP/2.0 (webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)/1.0 Profile Disallow: / User-agent: Vagabondo/1.x MT (webagent@wise-guys.nl) Disallow: / User-agent: Vagabondo/2.0 MT Disallow: / User-agent: Vagabondo/2.0 MT (webagent at wise-guys dot nl) Disallow: / User-agent: Vagabondo/2.0 MT (webagent@NOSPAMwise-guys.nl) Disallow: / User-agent: Vagabondo/3.0 (webagent at wise-guys dot nl) Disallow: / User-agent: Vakes/0.01 (Vakes; http://www.vakes.com/; search@vakes.com) Disallow: / User-agent: VeryGoodSearch.com.DaddyLongLegs Disallow: / User-agent: Verzamelgids/2.2 (http://www.verzamelgids.nl) Disallow: / User-agent: Vespa Crawler Disallow: / User-agent: VisBot/2.0 (Visvo.com Crawler; http://www.visvo.com/bot.html; bot@visvo.com) Disallow: / User-agent: Vision Research Lab image spider at vision.ece.ucsb.edu Disallow: / User-agent: Vortex/2.2 (+http://marty.anstey.ca/robots/vortex/) Disallow: / User-agent: W3SiteSearch Crawler_v1.1 http://www.w3sitesearch.de Disallow: / User-agent: WEP Search 00 Disallow: / User-agent: WFARC Disallow: / User-agent: WIRE/0.11 (Linux; i686; Bot,Robot,Spider,Crawler,aromano@cli.di.unipi.it) Disallow: / User-agent: WIRE/0.x (Linux; i686; Bot,Robot,Spider,Crawler) Disallow: / User-agent: WISEbot/1.0 (WISEbot@koreawisenut.com; http://wisebot.koreawisenut.com) Disallow: / User-agent: WSB WebCrawler V1.0 (Beta)- cl@cs.uni-dortmund.de Disallow: / User-agent: WSB, http://websearchbench.cs.uni-dortmund.de Disallow: / User-agent: WWWeasel Robot v1.00 (http://wwweasel.de) Disallow: / User-agent: Wavefire/0.8-dev (Wavefire; http://www.wavefire.com; info@wavefire.com) Disallow: / User-agent: Waypath Scout v2.x - info at waypath dot com Disallow: / User-agent: Waypath development crawler - info at waypath dot com Disallow: / User-agent: Web Snooper Disallow: / User-agent: Web-Bot V1.03 Disallow: / User-agent: WebAlta Crawler/1.2.1 (http://www.webalta.ru/bot.html) Disallow: / User-agent: WebCompass 2.0 Disallow: / User-agent: WebCorp/1.0 Disallow: / User-agent: WebFindBot(http://www.web-find.com) Disallow: / User-agent: WebRankSpider/1.37 (+http://ulm191.server4you.de/crawler/) Disallow: / User-agent: WebSearch.COM.AU/3.0.1 (The Australian Search Engine; http://WebSearch.COM.AU; Search@WebSearch.COM.AU) Disallow: / User-agent: WebSearchBench WebCrawler V1.0 (Beta)- Prof. Dr.-Ing. Christoph Lindemann- Universität Dortmund- cl@cs.uni-dortmund.de- http://websearchbench.cs.uni-dortmund.de/ Disallow: / User-agent: WebSearchBench WebCrawler v0.1(Experimental) Disallow: / User-agent: WebStat/1.0 (Unix; beta; 20040314) Disallow: / User-agent: WebVac (webmaster@pita.stanford.edu) Disallow: / User-agent: WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803 Disallow: / User-agent: WebWatcherMonitor/2.01 Disallow: / User-agent: WebarooBot (Webaroo Bot; http://64.124.122.252/feedback.html) Disallow: / User-agent: WebarooBot (Webaroo Bot; http://www.webaroo.com/rooSiteOwners.html) Disallow: / User-agent: Webclipping.com Disallow: / User-agent: Webglimpse 2.xx.x (http://webglimpse.net) Disallow: / User-agent: Weblog Attitude Diffusion 1.0 Disallow: / User-agent: WebsiteWorth v1.0 Disallow: / User-agent: Webspinne/1.0 webmaster@webspinne.de Disallow: / User-agent: Websquash.com (Add url robot) Disallow: / User-agent: Webster v0.3 ( http://webster.healeys.net/ ) Disallow: / User-agent: Webverzeichnis.de - Telefon: 01908 / 26005 Disallow: / User-agent: Wells Search II Disallow: / User-agent: WhizBang! Lab Disallow: / User-agent: Willow Internet Crawler by Twotrees V2.1 Disallow: / User-agent: WinHTTP Example/1.0 Disallow: / User-agent: WinkBot/0.06 (Wink.com search engine web crawler; http://www.wink.com/Wink:WinkBot; winkbot@wink.com) Disallow: / User-agent: WiseWire-Spider2 Disallow: / User-agent: WorldLight Disallow: / User-agent: Wotbox/alpha0.6 (bot@wotbox.com; http://www.wotbox.com) Disallow: / User-agent: Wotbox/alpha0.x.x (bot@wotbox.com; http://www.wotbox.com) Java/1.4.1_02 Disallow: / User-agent: Wusage/x.0@boutell.com Disallow: / User-agent: Wwlib/Linux Disallow: / User-agent: X-Crawler Disallow: / User-agent: Y!J-BSC/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Disallow: / User-agent: Y!J-SRD/1.0 Disallow: / User-agent: Y!J/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Disallow: / User-agent: Yahoo Pipes 1.0 Disallow: / User-agent: Yahoo! Mindset Disallow: / User-agent: Yahoo-Blogs/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/us/ysearch/crawling/crawling-02.html ) Disallow: / User-agent: Yahoo-MMAudVid/1.0 (mms dash mmaudvidcrawler dash support at yahoo dash inc dot com) Disallow: / User-agent: Yahoo-MMAudVid/2.0(mms dash mm aud vid crawler dash support at yahoo dash inc.com ;Mozilla 4.0 compatible; MSIE 7.0;Windows NT 5.0; .NET CLR 2.0) Disallow: / User-agent: Yahoo-MMCrawler/3.x (mm dash crawler at trd dot overture dot com) Disallow: / User-agent: Yahoo-Test/4.0 Disallow: / User-agent: Yahoo-VerticalCrawler-FormerWebCrawler/3.9 crawler at trd dot overture dot com; http://www.alltheweb.com/help/webmaster/crawler Disallow: / User-agent: YahooFeedSeeker/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://publisher.yahoo.com/rssguide) Disallow: / User-agent: YahooSeeker-Testing/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://search.yahoo.com/) Disallow: / User-agent: YahooSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/us/shop/merchant/) Disallow: / User-agent: YahooSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5; http://search.yahoo.com/yahooseeker.html) Disallow: / User-agent: YahooSeeker/1.1 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/us/shop/merchant/) Disallow: / User-agent: YahooSeeker/CafeKelsa-dev (compatible; Konqueror/3.2; FreeBSD ;cafekelsa-dev-webmaster@yahoo-inc.com ) Disallow: / User-agent: YahooSeeker/bsv3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/us/ysearch/crawling/crawling-02.html ) Disallow: / User-agent: Yandex/1.01.001 (compatible; Win16; I) Disallow: / User-agent: Yeti Disallow: / User-agent: Yeti/0.01 (nhn/1noon, yetibot@naver.com, check robots.txt daily and follows it) Disallow: / User-agent: YodaoBot/1.0 (http://www.yodao.com/help/webmaster/spider/; ) Disallow: / User-agent: YottaCars_Bot/4.12 (+http://www.yottacars.com) Car Search Engine Disallow: / User-agent: YottaShopping_Bot/4.12 (+http://www.yottashopping.com) Shopping Search Engine Disallow: / User-agent: ZBot/1.00 (icaulfield@zeus.com) Disallow: / User-agent: ZIPPPCVS/0.xx (ZipppBot/.xx;http://www.zippp.net; webmaster@zippp.net) Disallow: / User-agent: Zao-Crawler Disallow: / User-agent: Zao-Crawler 0.2b Disallow: / User-agent: Zao/0.1 (http://www.kototoi.org/zao/) Disallow: / User-agent: ZeBot_lseek.net (bot@ze.bz) Disallow: / User-agent: ZeBot_www.ze.bz (ze.bz@hotmail.com) Disallow: / User-agent: Zearchit Disallow: / User-agent: Zeus ThemeSite Viewer Webster Pro V2.9 Win32 Disallow: / User-agent: Zeus xxxxx Webster Pro V2.9 Win32 Disallow: / User-agent: Zeusbot/0.07 (Ulysseek's web-crawling robot; http://www.zeusbot.com; agent@zeusbot.com) Disallow: / User-agent: ZipppBot/0.xx (ZipppBot; http://www.zippp.net; webmaster@zippp.net) Disallow: / User-agent: Zippy v2.0 - Zippyfinder.com Disallow: / User-agent: ZoomSpider - wrensoft.com Disallow: / User-agent: ZyBorg/1.0 (ZyBorg@WISEnut.com; http://www.WISEnut.com) Disallow: / User-agent: aardvark-crawler Disallow: / User-agent: abot/0.1 (abot; http://www.abot.com; abot@abot.com) Disallow: / User-agent: accoona Disallow: / User-agent: agadine/1.x.x (+http://www.agada.de) Disallow: / User-agent: aipbot/1.0 (aipbot; http://www.aipbot.com; aipbot@aipbot.com) Disallow: / User-agent: aipbot/2-beta (aipbot dev; http://aipbot.com; aipbot@aipbot.com) Disallow: / User-agent: amibot Disallow: / User-agent: antibot-V1.1.5/i586-linux-2.2 Disallow: / User-agent: appie 1.1 (www.walhello.com) Disallow: / User-agent: archive.org_bot Disallow: / User-agent: ask.24x.info Disallow: / User-agent: asked/Nutch-0.8 (web crawler; http://asked.jp; epicurus at gmail dot com) Disallow: / User-agent: asterias/2.0 Disallow: / User-agent: atSpider/1.0 Disallow: / User-agent: augurfind Disallow: / User-agent: augurnfind V-1.x Disallow: / User-agent: autoemailspider Disallow: / User-agent: autowebdir 1.1 (www.autowebdir.com) Disallow: / User-agent: axadine/ (Axadine Crawler; http://www.axada.de/; ) Disallow: / User-agent: bdcindexer_2.6.2 (research@bdc) Disallow: / User-agent: beautybot/1.0 (+http://www.uchoose.de/crawler/beautybot/) Disallow: / User-agent: blogWatcher_Spider/0.1 (http://www.lr.pi.titech.ac.jp/blogWatcher/) Disallow: / User-agent: blogsearchbot-pumpkin-3 Disallow: / User-agent: boitho.com-dc/0.xx (http://www.boitho.com/dcbot.html) Disallow: / User-agent: boitho.com-robot/1.x Disallow: / User-agent: boitho.com-robot/1.x (http://www.boitho.com/bot.html) Disallow: / User-agent: bumblebee@relevare.com Disallow: / User-agent: bwh3_user_agent Disallow: / User-agent: carleson/1.0 Disallow: / User-agent: ccubee/x.x Disallow: / User-agent: cfetch/1.0 Disallow: / User-agent: collage.cgi/1.xx Disallow: / User-agent: combine/0.0 Disallow: / User-agent: cometrics-bot, http://www.cometrics.de Disallow: / User-agent: cosmos/0.8_(robot@xyleme.com) Disallow: / User-agent: cosmos/0.9_(robot@xyleme.com) Disallow: / User-agent: crawler@alexa.com Disallow: / User-agent: csci_b659/0.13 Disallow: / User-agent: dCSbot/1.1 Disallow: / User-agent: daypopbot/0.x Disallow: / User-agent: dbDig(http://www.prairielandconsulting.com) Disallow: / User-agent: de.searchengine.comBot 1.2 (http://de.searchengine.com/spider) Disallow: / User-agent: deepak-USC/ISI Disallow: / User-agent: dev-spider2.searchpsider.com/1.3b Disallow: / User-agent: disco/Nutch-0.9 (experimental crawler; www.discoveryengine.com; disco-crawl@discoveryengine.com) Disallow: / User-agent: disco/Nutch-1.0-dev (experimental crawler; www.discoveryengine.com; disco-crawl@discoveryengine.com) Disallow: / User-agent: dloader(NaverRobot)/1.0 Disallow: / User-agent: dodgebot/experimental Disallow: / User-agent: dtSearchSpider Disallow: / User-agent: e-SocietyRobot(http://www.yama.info.waseda.ac.jp/~yamana/es/) Disallow: / User-agent: e-sense 1.0 ea(www.vigiltech.com/esensedisclaim.html) Disallow: / User-agent: eApolloBot/2.0 (compatible; heritrix/2.0.0-SNAPSHOT-20071024.170148 +http://www.eapollo-opto.com) Disallow: / User-agent: eStyleSearch 4 (compatible; MSIE 6.0; Windows NT 5.0) Disallow: / User-agent: egothor/3.0a (+http://www.xdefine.org/robot.html) Disallow: / User-agent: ejupiter.com Disallow: / User-agent: elfbot/1.0 (+http://www.uchoose.de/crawler/elfbot/) Disallow: / User-agent: envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.php) Disallow: / User-agent: envolk[ITS]spider/1.6(+http://www.envolk.com/envolkspider.html) Disallow: / User-agent: eseek-larbin_2.6.2 (crawler@exactseek.com) Disallow: / User-agent: eventax/1.3 (eventax; http://www.eventax.de/; info@eventax.de) Disallow: / User-agent: exactseek-crawler-2.63 (crawler@exactseek.com) Disallow: / User-agent: exactseek-pagereaper-2.63 (crawler@exactseek.com) Disallow: / User-agent: exactseek.com Disallow: / User-agent: exooba crawler/exooba crawler (crawler for exooba.com; http://www.exooba.com/; info at exooba dot com) Disallow: / User-agent: exooba/exooba crawler (exooba; exooba) Disallow: / User-agent: factbot : http://www.factbites.com/robots Disallow: / User-agent: fastbot crawler beta 2.0 (+http://www.fastbot.de) Disallow: / User-agent: favo.eu crawler/0.6 (http://www.favo.eu) Disallow: / User-agent: fileboost.net/1.0 (+http://www.fileboost.net) Disallow: / User-agent: findlinks/x.xxx (+http://wortschatz.uni-leipzig.de/findlinks/) Disallow: / User-agent: firstsbot Disallow: / User-agent: flunky Disallow: / User-agent: gamekitbot/1.0 (+http://www.uchoose.de/crawler/gamekitbot/) Disallow: / User-agent: gazz/x.x (gazz@nttrd.com) Disallow: / User-agent: generic_crawler/01.0217/ Disallow: / User-agent: genieBot (http://64.5.245.11/faq/faq.html) Disallow: / User-agent: geniebot wgao@genieknows.com Disallow: / User-agent: gigabaz/3.1x (baz@gigabaz.com; http://gigabaz.com/gigabaz/) Disallow: / User-agent: gonzo1[P] +http://www.suchen.de/popups/faq.jsp Disallow: / User-agent: gonzo2[P] +http://www.suchen.de/faq.html Disallow: / User-agent: great-plains-web-spider/flatlandbot (Flatland Industries Web Spider; http://www.flatlandindustries.com/flatlandbot.php; jason@flatlandindustries.com) Disallow: / User-agent: grub crawler(http://www.grub.org) Disallow: / User-agent: grub-client Disallow: / User-agent: gsa-crawler (Enterprise; GID-01422; jplastiras@google.com) Disallow: / User-agent: gsa-crawler (Enterprise; GID-01742;gsatesting@rediffmail.com) Disallow: / User-agent: gsa-crawler (Enterprise; GIX-02057; dm@enhesa.com) Disallow: / User-agent: gsa-crawler (Enterprise; GIX-03519; cknuetter@stubhub.com) Disallow: / User-agent: gsa-crawler (Enterprise; GIX-0xxxx; enterprise-training@google.com) Disallow: / User-agent: hbtronix.spider.2 -- http://hbtronix.de/spider.php Disallow: / User-agent: holmes/3.11 (http://morfeo.centrum.cz/bot) Disallow: / User-agent: holmes/3.9 (onet.pl) Disallow: / User-agent: holmes/3.xx (OnetSzukaj/5.0; +http://szukaj.onet.pl) Disallow: / User-agent: holmes/x.x Disallow: / User-agent: htdig/3.1.6 (http://computerorgs.com) Disallow: / User-agent: htdig/3.1.6 (unconfigured@htdig.searchengine.maintainer) Disallow: / User-agent: htdig/3.1.x (root@localhost) Disallow: / User-agent: http://Ask.24x.Info/ (http://narres.it/) Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler [rc1.wf.ibm.com] Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler [wf216] Disallow: / User-agent: http://www.istarthere.com_spider@istarthere.com Disallow: / User-agent: http://www.monogol.de Disallow: / User-agent: http://www.trendtech.dk/spider.asp) Disallow: / User-agent: i1searchbot/2.0 (i1search web crawler; http://www.i1search.com; crawler@i1search.com) Disallow: / User-agent: iCCrawler (http://www.iccenter.net) Disallow: / User-agent: iFeed.jp/2.0 (www.psychedelix.com/agents/agents.rss; 0 subscribers) Disallow: / User-agent: ia_archiver Disallow: / User-agent: ia_archiver-web.archive.org Disallow: / User-agent: ia_archiver/1.6 Disallow: / User-agent: iaskspider2 (iask@staff.sina.com.cn) Disallow: / User-agent: ichiro/x.0 (http://help.goo.ne.jp/door/crawler.html) Disallow: / User-agent: ichiro/x.0 (ichiro@nttr.co.jp) Disallow: / User-agent: icsbot-0.1 Disallow: / User-agent: ideare - SignSite/1.x Disallow: / User-agent: igdeSpyder (compatible; igde.ru; +http://igde.ru/doc/tech.html) Disallow: / User-agent: ilial/Nutch-0.9 (Ilial, Inc. is a Los Angeles based Internet startup company. For more information please visit http://www.ilial.com/crawler; http://www.ilial.com/crawler; crawl@ilial.com) Disallow: / User-agent: ilial/Nutch-0.9-dev Disallow: / User-agent: iltrovatore-setaccio/1.2-dev (spidering; http://www.iltrovatore.it/aiuto/.....) Disallow: / User-agent: info@pubblisito.com- (http://www.pubblisito.com) il Sud dei Motori di Ricerca Disallow: / User-agent: infoConveraCrawler/0.8 ( http://www.authoritativeweb.com/crawl) Disallow: / User-agent: k2spider Disallow: / User-agent: kalooga/kalooga-4.0-dev-datahouse (Kalooga; http://www.kalooga.com; info@kalooga.com) Disallow: / User-agent: kinja-imagebot (http://www.kinja.com/) Disallow: / User-agent: kinjabot (http://www.kinja.com) Disallow: / User-agent: kuloko-bot/0.x Disallow: / User-agent: kulokobot www.kuloko.com kuloko@backweave.com Disallow: / User-agent: kulturarw3/0.1 Disallow: / User-agent: larbin (samualt9@bigfoot.com) Disallow: / User-agent: larbin_2.1.1 larbin2.1.1@somewhere.com Disallow: / User-agent: larbin_2.2.0 (crawl@compete.com) Disallow: / User-agent: larbin_2.2.1_de_Viennot (Laurent.Viennot@inria.fr) Disallow: / User-agent: larbin_2.2.2 (sugayama@lab7.kuis.kyoto-u.ac.jp) Disallow: / User-agent: larbin_2.2.2_guillaume (guillaume@liafa.jussieu.fr) Disallow: / User-agent: larbin_2.6.0 (larbin2.6.0@unspecified.mail) Disallow: / User-agent: larbin_2.6.1 (larbin2.6.1@unspecified.mail) Disallow: / User-agent: larbin_2.6.2 (hamasaki@grad.nii.ac.jp) Disallow: / User-agent: larbin_2.6.2 (larbin2.6.2@unspecified.mail) Disallow: / User-agent: larbin_2.6.2 (listonATccDOTgatechDOTedu) Disallow: / User-agent: larbin_2.6.2 (pimenas@systems.tuc.gr) Disallow: / User-agent: larbin_2.6.2 (tom@lemurconsulting.com) Disallow: / User-agent: larbin_2.6.2 (vitalbox1@hotmail.com) Disallow: / User-agent: larbin_2.6.3 (ltaa_web_crawler@groupes.epfl.ch) Disallow: / User-agent: larbin_2.6.3 (wgao@genieknows.com) Disallow: / User-agent: larbin_2.6.3_for_(http://cosco.hiit.fi/search/) tsilande@hiit.fi Disallow: / User-agent: larbin_2.6_basileocaml (basile.starynkevitch@cea.fr) Disallow: / User-agent: larbin_devel (http://pauillac.inria.fr/~ailleret/prog/larbin/) Disallow: / User-agent: lawinfo-crawler/Nutch-0.9-dev (Crawler for lawinfo.com pages; http://www.lawinfo.com; webmaster@lawinfo.com) Disallow: / User-agent: libWeb/clsHTTP -- hiongun@kt.co.kr Disallow: / User-agent: libwww-perl/5.41 Disallow: / User-agent: libwww-perl/5.45 Disallow: / User-agent: libwww-perl/5.48 Disallow: / User-agent: libwww-perl/5.50 Disallow: / User-agent: libwww-perl/5.52 FP/2.1 Disallow: / User-agent: libwww-perl/5.52 FP/4.0 Disallow: / User-agent: libwww-perl/5.53 Disallow: / User-agent: libwww-perl/5.63 Disallow: / User-agent: libwww-perl/5.64 Disallow: / User-agent: libwww-perl/5.65 Disallow: / User-agent: libwww-perl/5.800 Disallow: / User-agent: libwww/5.3.2 Disallow: / User-agent: linkbot Disallow: / User-agent: linknzbot Disallow: / User-agent: lmspider (lmspider@scansoft.com) Disallow: / User-agent: lwp-trivial/1.32 Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: mailto:webcraft@bea.com Disallow: / User-agent: mammoth/1.0 ( http://www.sli-systems.com/) Disallow: / User-agent: maxomobot/dev-20051201 (maxomo; http://67.102.134.34:4047/MAXOMO/MAXOMObot.html; maxomobot@maxomo.com) Disallow: / User-agent: metatagsdir/0.7 (+http://metatagsdir.com/directory/) Disallow: / User-agent: moget/x.x (moget@goo.ne.jp) Disallow: / User-agent: mogimogi/1.0 Disallow: / User-agent: moiNAG 0.02 Disallow: / User-agent: mozDex/0.xx-dev (mozDex; http://www.mozdex.com/en/bot.html; spider@mozdex.com) Disallow: / User-agent: msnbot-Products/1.0 (+http://search.msn.com/msnbot.htm) Disallow: / User-agent: msnbot-media/1.0 (+http://search.msn.com/msnbot.htm) Disallow: / User-agent: msnbot/x.xx ( http://search.msn.com/msnbot.htm) Disallow: / User-agent: multicrawler ( http://sw.deri.org/2006/04/multicrawler/robots.html) Disallow: / User-agent: nabot_1.0 Disallow: / User-agent: noxtrumbot/1.0 (crawler@noxtrum.com) Disallow: / User-agent: noyona_0_1 Disallow: / User-agent: nsyght.com/Nutch-1.0-dev (nsyght.com; Nsyght.com) Disallow: / User-agent: nsyght.com/Nutch-x.x (nsyght.com; search.nsyght.com) Disallow: / User-agent: nttdirectory_robot/0.9 (super-robot@super.navi.ocn.ne.jp) Disallow: / User-agent: nuSearch Spider www.nusearch.com (compatible; MSIE 4.01) Disallow: / User-agent: nutchsearch/Nutch-0.9 (Nutch Search 1.0; herceg_novi at yahoo dot com) Disallow: / User-agent: oBot ((compatible;Win32)) Disallow: / User-agent: obidos-bot (just looking for books.) Disallow: / User-agent: ozelot/2.7.3 (Search engine indexer; www.flying-cat.de/ozelot; ozelot@flying-cat.de) Disallow: / User-agent: page_verifier (http://www.securecomputing.com/goto/pv) Disallow: / User-agent: parallelContextFocusCrawler1.1parallelContextFocusCrawler1.1 Disallow: / User-agent: pd02_1.0.0 pd02_1.0.0@dzimi@post.sk Disallow: / User-agent: pipeLiner/0.3a (PipeLine Spider;http://www.pipeline-search.com/webmaster.html; webmaster'at'pipeline-search.com) Disallow: / User-agent: pipeLiner/0.xx (PipeLine Spider; http://www.pipeline-search.com/webmaster.html) Disallow: / User-agent: polybot 1.0 (http://cis.poly.edu/polybot/) Disallow: / User-agent: potbot 1.0 Disallow: / User-agent: psbot/0.1 (+http://www.picsearch.com/bot.html) Disallow: / User-agent: psycheclone Disallow: / User-agent: pulseBot (pulse Web Miner) Disallow: / User-agent: rabaz (rabaz at gigabaz dot com) Disallow: / User-agent: ramBot xtreme x.x Disallow: / User-agent: rico/0.1 Disallow: / User-agent: sait/Nutch-0.9 (SAIT Research; http://www.samsung.com) Disallow: / User-agent: savvybot/0.2 Disallow: / User-agent: schwarzmann.biz-Spider_for_paddel.org+(http://www.innerprise.net/usp-spider.asp) Disallow: / User-agent: scooter-venus-3.0.vns Disallow: / User-agent: scoutmaster Disallow: / User-agent: search.ch V1.4 Disallow: / User-agent: search.ch V1.4.2 (spiderman@search.ch; http://www.search.ch) Disallow: / User-agent: searchbot admin@google.com Disallow: / User-agent: semanticdiscovery/0.x Disallow: / User-agent: silk/1.0 Disallow: / User-agent: silk/1.0 (+http://www.slider.com/silk.htm)/3.7 Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: sitecheck.internetseer.com (For more info see: http://sitecheck.internetseer.com) Disallow: / User-agent: smartwit.com Disallow: / User-agent: snap.com beta crawler v0 Disallow: / User-agent: sogou develop spider Disallow: / User-agent: sogou spider Disallow: / User-agent: sohu agent Disallow: / User-agent: sohu-search Disallow: / User-agent: speedfind ramBot xtreme 8.1 Disallow: / User-agent: spider.batsch.com Disallow: / User-agent: spider.yellopet.com - www.yellopet.com Disallow: / User-agent: sportsuchmaschine.de-Robot (Version: 1.02- powered by www.sportsuchmaschine.de) Disallow: / User-agent: sproose/0.1-alpha (sproose crawler; http://www.sproose.com/bot.html; crawler@sproose.com) Disallow: / User-agent: stat statcrawler@gmail.com Disallow: / User-agent: suchbaer.de Disallow: / User-agent: suchbaer.de (CrawlerAgent v0.103) Disallow: / User-agent: suchbot Disallow: / User-agent: suchpadbot/1.0 (+http://www.suchpad.de) Disallow: / User-agent: suzuran Disallow: / User-agent: tags2dir.com/0.8 (+http://tags2dir.com/directory/) Disallow: / User-agent: targetblaster.com/0.9k Disallow: / User-agent: teoma_agent1 Disallow: / User-agent: teomaagent crawler-admin@teoma.com Disallow: / User-agent: teomaagent1 [crawler-admin@teoma.com] Disallow: / User-agent: terraminds-bot/1.0 (support@terraminds.de) Disallow: / User-agent: thumbshots-de-Bot (Version: 1.02- powered by www.thumbshots.de) Disallow: / User-agent: timboBot/0.9 http://www.breakingblogs.com/timbo_bot.html Disallow: / User-agent: tivraSpider/1.0 (crawler@tivra.com) Disallow: / User-agent: traazibot/testengine (+http://www.traazi.de) Disallow: / User-agent: updated/0.1-alpha (updated crawler; http://www.updated.com; crawler@updated.com) Disallow: / User-agent: updated/0.1beta (updated.com; http://www.updated.com; crawler@updated.om) Disallow: / User-agent: user-agent=Mozilla/3.01Gold Disallow: / User-agent: versus 0.2 (+http://versus.integis.ch) Disallow: / User-agent: versus crawler eda.baykan@epfl.ch Disallow: / User-agent: verzamelgids.nl - Networking4all Bot/x.x Disallow: / User-agent: voyager-hc/1.0 Disallow: / User-agent: voyager/1.0 Disallow: / User-agent: vspider Disallow: / User-agent: vspider/3.x Disallow: / User-agent: wadaino.jp-crawler 0.2 (http://wadaino.jp/) Disallow: / User-agent: web2express.org/Nutch-0.9-dev (leveled playing field; http://web2express.org/; info at web2express.org) Disallow: / User-agent: webGobbler/1.x.x Disallow: / User-agent: webbandit/4.xx.0 Disallow: / User-agent: webcollage/1.xx Disallow: / User-agent: webcrawl.net Disallow: / User-agent: webmeasurement-bot, http://rvs.informatik.uni-leipzig.de Disallow: / User-agent: whatUseek_winona/3.0 Disallow: / User-agent: worio heritrix bot (+http://worio.com/) Disallow: / User-agent: woriobot ( http://www.worio.com/) Disallow: / User-agent: wume_crawler/1.1 (http://wume.cse.lehigh.edu/~xiq204/crawler/) Disallow: / User-agent: www.arianna.it Disallow: / User-agent: wwwster/1.x (Beta- mailto:gue@cis.uni-muenchen.de) Disallow: / User-agent: xirq/0.1-beta (xirq; http://www.xirq.com; xirq@xirq.com) Disallow: / User-agent: xyro_(xcrawler@cosmos.inria.fr) Disallow: / User-agent: yacy (www.yacy.net; v20040602; i386 Linux 2.4.26-gentoo-r13; java 1.4.2_06; MET/en) Disallow: / User-agent: yacybot (x86 Windows XP 5.1; java 1.5.0_06; Europe/de) yacy.net Disallow: / User-agent: yarienavoir.net/0.2 Disallow: / User-agent: yggdrasil/Nutch-0.9 (yggdrasil biorelated search engine; www dot biotec dot tu minus dresden do de slash schroeder; heiko dot dietze at biotec dot tu minus dresden dot de) Disallow: / User-agent: yoofind/yoofind-0.1-dev (yoono webcrawler; http://www.yoono.com ; MyEmail) Disallow: / User-agent: yoogliFetchAgent/0.1 Disallow: / User-agent: yoono/1.0 web-crawler/1.0 Disallow: / User-agent: zedzo.digest/0.1 (http://www.zedzo.com/) Disallow: / User-agent: zermelo Mozilla/5.0 compatible; heritrix/1.12.1 (+http://www.powerset.com) [email:crawl@powerset.com,email:paul@page-store.com] Disallow: / User-agent: zerxbot/Version 0.6 libwww-perl/5.79 Disallow: / User-agent: zspider/0.9-dev http://feedback.redkolibri.com/ Disallow: / User-agent: * Disallow: /button Disallow: /contact Disallow: /images Disallow: /moving_water Disallow: /rose Disallow: /templates