A robots.txt file is a plain text file at a site's root (zeo.org/robots.txt) that gives crawlers instructions about which parts of the site they're allowed to request, using Allow and Disallow directives, before those crawlers request anything else on the site.
It's a voluntary standard, not an enforced restriction: well-behaved crawlers like Googlebot check and respect it, but nothing technically stops a bot from ignoring it entirely, which means robots.txt is a crawl-management tool rather than a security or access-control mechanism. It's not the tool for keeping a page out of search results either; that's what a noindex tag does. A single mistake in robots.txt, an overly broad Disallow rule, an accidental block on the entire site during a redesign, can cut off crawling to large sections of a site or all of it at once, silently, since the block itself produces no visible error for anyone who isn't specifically checking. That combination of high leverage and easy-to-miss mistakes is exactly why it's worth testing changes to this file carefully before pushing them live.
