Using Robots.txt File
Overview of Robots.txt
A robots file, named “robots.txt” is simply an “Access” file stored in your web server’s root directory. Creating a robots.txt file doesn’t improve your search engine ranking, but it does provide robots, or web crawlers, with specific information concerning which files you will not allow to be crawled and indexed in the search engines.
For example, if we had an area on our web site, such as a family blog that we didn’t the search engines to show the pages in the search engine results pages, or “serps,” we could specify in the robots.txt file, to exclude our blog directory.
We would do this by adding these two lines of code to the robots.txt file.
User-agent: *
Disallow: /blog/
If we want our entire web site crawled by the bots, we would simply either upload a blank robots.txt file into our web servers root directory our we would change the “Disallow:” to have nothing after it as in the example below:
User-agent: *
Disallow:
I have provide the following links for more information on the robots.txt file and it’s uses:
http://www.robotstxt.org/wc/robots.html
http://en.wikipedia.org/wiki/Robots.txt
Here is a link to a robots.txt file generator:
http://www.mcanerin.com/EN/search-engine/robots-txt.asp