Welcome, Guest. [ Log In ]
Question   Robots.txt
Search KBase


Top in this Area:
1. Robots.txt

 
Robots.txt
By creating a "robots.txt" file at the root of your site/domain, you can tell search engines what content on your site they should and should not index. This can be helpful, for example, if you want to keep a portion of your site out of the Google search engine index.

Note that this file is meant as a suggestion to compliant search engines only, and does not prevent all search engines (or other similar tools, such as email/content scrapers) from accessing the content or making it available. However, most of the major engines respect robots.txt directives.

A good resource for information on the actual creation and format of robots.txt files can be found here:

     http://www.robotstxt.org/

Last updated: Dec 02, 2004.