How Robots.Txt Works

The file robots.txt is used to inform the search engines about the pages of your site that you do not want indexed in the search engines. I will show you in detail how to create such a file and why it is useful.

Presentation.

The file robots.txt is a small text file that contains information on destination robots which route your Internet site. Generally it is useful for robots search engines, because it indicates the pages to index or to the contrary deny them access to some pages (or parts) of the site.

The use of such a file has become trite and search engines start by checking if such a file exists on your site even before starting the analysis of it.

Benefits.

Most websites big enough owns a private party that they do not want to disclose to the search engine. The file of exclusion of robots, then indicates what robots (a robot particular or all the robots) should exclude what pages. It can also be useful if you do not want to index some pages of a building site.

This method is more appropriate then using the meta tag "robots". This tag in question may look like this: Code:

< meta name="robots" content="none" / >The problem is that with this tag is that you lose room on each of your page compared to the real content. In addition, if you must protect all pages of a folder (example, your file "admin"), it will be a bit long to go change all the pages one by one. I should also note that the meta tags are a little old and that the file robots is more recent.

Therefore, using a file which brings together all the rules of indexing your site is more convenient and more readable.

Nevertheless, you should be aware that this file is only an indication for the honest robots (those involved in general to big sites). Some malicious robots may not read at all this file. It is especially true for robots that scans your site in search of e-mail to spam you.

There is also the old robots not too efficient and not sufficiently developed to analyze this file of exclusion of engines.

How to put it in place.