SEO · Website glossary

robots.txt

Quick answer · What is robots.txt?

A file at a site's root that gives compliant crawlers rules about which URLs they may request.

IN PRACTICE

robots.txt: an example

A site publishes crawler-specific disallow rules for selected paths.

KEEP IN MIND

A crawl block is not a reliable way to keep a URL out of search results or to secure private content.

Put the idea to work

Frequently asked questions

What is a practical example of robots.txt?

A site publishes crawler-specific disallow rules for selected paths.

Can robots.txt reliably keep a URL out of search or secure private content?

A crawl block is not a reliable way to keep a URL out of search results or to secure private content.

Which concepts should I read alongside robots.txt?

noindex: A robots directive asking supporting search engines not to include a page in search results.
Crawling: The discovery and fetching of pages or resources by an automated crawler.