As you may, know the robots.txt is a tool used by the search engines to see which pages not to index. It is useful if you have sensitive information or other pages you don't wish to get indexed - that is all it does.
It is simply a text file names robots.txt located in the root.
It contains a line for user-agent and then what to disallow or allow.
Example:
User-agent: *
Disallow: /password.html
Disallow: /temp/
As the * is a wildcard the above means that all user-agents will index all pages except the file password.html and the directory temp and it's content.
Here is another example when disallowing all from a bad spider.
User-agent: badspider
Disallow: /
