Robots.txt

Robots.txt meaning in English

Robots.txt

What is a Robots.txt file?

Robots.txt files are used as a communication method to control which pages can be crawled by bots.

It is possible to allow and disallow all, or specific search engines/bots on areas of a website.

The robots.txt file should be found at the root of a website, for example https://www.example.com/robots.txt.

Robots.txt files can also include a references to sitemaps.

Example

The following example robots.txt file contains a rule blocking search engines from a specific page, and a reference to an XML sitemap file.

User-agent: *
Disallow: /blocked-page/

Sitemap: https://www.domain.com/sitemap.xmlCode language: plaintext (plaintext)

Related

Resources

Citation URL

https://www.searchcandy.uk/seo/seo-glossary/robots-txt/

Last modified: Created:

What is this content? This article is part of an SEO glossary and reference guide created by Search Candy - an SEO consultancy based in the UK. The Search Candy team is committed to providing content that adheres to the highest editorial standards. The date this article was created and last checked for accuracy is posted above. To reuse this content please get in touch via our contact form.
chevron-down