Googlebot/2.1+(+http://www.google.com/bot.html)  // Google's Spider
Mozilla/5.0+(compatible;+Yahoo!+Slurp;+http://help.yahoo.com/help/us/ysearch/slurp)  //Yahoo's Spider

These 2 are nice spiders as they identify themselves appropriately. User Agents do not have to identify themselves correctly, so sometimes spotting them (if you are looking for them) can be difficult. Most play nice – yea!

I, Robot(.txt)

The robots.txt that I have creates is as follows:

User-agent: *
Disallow: /_css/
Disallow: /_images/
Disallow: /_scripts/
Disallow: /_test/
Disallow: /data/stuff/*.htm
Disallow: /mm2css/
Disallow: /mm2img/
Disallow: /mm2script/

There is not much to it. It is a really simple process to create one. The first part of a robots.txt is the User-Agent line to which you specify for which User Agents the following rules apply to. We Specified * which means all all user agents. We can specify rules for specific User Agents such as Googlebot by the following: User-Agent:Googlebot, and the rules following it would be picked up by Googlebot. Nothing forces a Bot or Spider to follow the rules that is specified in the robots.txt file. They are followed by choice.

The next statements are exclusion statements. We tell the User Agents what directories to exclude from indexing. Some bots will support file level exclusions. I use both here. I have disallowed all image, scripting and CSS as they cannot be indexed by robots, so I will just save them the time. I have also excluded _test directories as they do not have indexable data. The last thing I excluded was *.htm in the Stuff Directory. We do not want people going to stuff. The index.html is still indexed for the stuff directory because it has the .html extension, but the “subordinate” pages are not as they have the .htm extension. The robots.txt file is looked for at the site’s root.

Other Means to Restrict Access

No matter what you put in your robots.txt file a bot can always have aceess to your resources as the robots.txt file is not forced on user agents. If you want to ensure user agents do not have access to your resources secure it with permissions.

To reduce the chance that a bot will look to index your site you can try to ensure that there are no links to it. If there are no links then a bot will have very little reason to index it.