Welcome, Guest. [ Log In ]
Question   Robots.txt
Search KBase

Top in this Area:
1. Robots.txt

By creating a "robots.txt" file at the root of your site/domain, you can tell search engines what content on your site they should and should not index. This can be helpful, for example, if you want to keep a portion of your site out of the Google search engine index.

Note that this file is meant as a suggestion to compliant search engines only, and does not prevent all search engines (or other similar tools, such as email/content scrapers) from accessing the content or making it available. However, most of the major engines respect robots.txt directives.

A good resource for information on the actual creation and format of robots.txt files can be found here:


Last updated: Dec 02, 2004.