[Bot] TheRarestParser (robots.txt)
TheRarestParser is my bot which goes out collecting words used in web pages for “The Rarest Words” project.
Read about the project here or on TechCrunch (TheRarestWords: Intriguing Semantic SEO Project from Russia) or see it in action by visiting
http://therarestwords.com/your-site-url.com
(replace your-site.com with your site’s url - for your convenience it’s sent in referrer field during the bot’s visit).
Current version: 0.4b (and 0.2a)
User-agent string: TheRarestParser/0.4b (http://rarestblog.com/therarestparser/)
IPs: 72.232.237.202
Referrer: link to your page
As of 19 May 2008 the robot obeys robots.txt, so if you want to block it from visiting your site, just add this to your robots.txt:
User-agent: TheRarestParser Disallow: /
If you are having a problem with my bot - send me the problem description to rarestwords@mail.ru