Robots, Spiders, and other Crawlers… Oh My!!
Robots are automated programs spawned by search engines to find and index content. Each search engine has it’s own robot, which are sometimes referred to as spiders, that (usually) leave a customized User Agent String to identify itself to web servers. Here are example user agent strings found on our server logs:
Googlebot/2.1+(+http://www.google.com/bot.html) // Google’s [...]
