robots.txt
BeginnerThe plain-text file at a site's root that tells crawlers which paths they may visit — a voluntary standard that separates polite bots from rude ones.
In depth
robots.txt is a plain-text file served at a website's root (example.com/robots.txt) that states the owner's crawling rules: which bots are addressed, which paths they're asked to avoid, and often where the sitemap lives. It is the web's oldest etiquette protocol — the Robots Exclusion Protocol, dating to 1994 and formalized as RFC 9309 in 2022.
How the rules read
Directives are grouped by User-agent, matched to the crawler's declared identity: Disallow asks a bot to skip a path prefix, Allow carves exceptions, wildcards refine patterns, and Sitemap points to the URL inventory. A group addressed to * applies to any bot without its own section.
What it is — and isn't
- It's a request, not a lock: nothing technically stops a crawler from ignoring it. Search engines comply scrupulously; the long tail of scrapers varies from respectful to oblivious.
- It's not access control: disallowed paths remain publicly fetchable, and listing sensitive URLs in robots.txt actually advertises them. Real protection requires authentication.
- It's legally meaningful anyway: courts and platforms increasingly treat robots.txt as evidence of the owner's expressed wishes — ignoring it weakens a scraper's legal posture even where scraping itself is lawful. The AI era sharpened this: blocking AI-training crawlers via robots.txt has become a mainstream publisher stance.
Professional practice
Check robots.txt before crawling any target — it's one HTTP request that tells you the owner's rules, reveals sitemaps that make crawling cheaper, and keeps your operation defensible. Respecting it is both ethics and self-interest.
Examples
- An online store disallows /cart and /checkout for all bots while pointing crawlers to its product sitemap.
- A publisher adds a robots.txt group blocking an AI company's training crawler by name.
- A scraping team reviews a target's robots.txt and excludes disallowed paths from its crawl plan.
Common use cases
FAQs
Not a law by itself, but far from meaningless: it documents the owner's expressed wishes, and courts, platforms, and terms-of-service disputes weigh it. Complying keeps a scraping operation on defensible ground; ignoring it invites both blocks and liability arguments.
No — it's voluntary. Well-run crawlers honor it; bad actors ignore it entirely. Enforcement comes from the site's anti-bot stack, not the file. Owners should treat robots.txt as policy declaration, never as security.
Never — the file is public, so disallowing /admin advertises exactly where the sensitive paths are. Use authentication for anything private; use robots.txt only to manage well-behaved crawler traffic on public content.