The Disallow directive in a robots.txt file tells crawlers which pages or folders they shouldn't request at all. A line like Disallow: /admin/ tells well-behaved bots to skip everything under that path, freeing up crawl budget for pages that actually matter.
Disallow blocks crawling, not indexing, a distinction that trips people up often: a disallowed page can still show up in search results, usually with no description, if other sites link to it, because the block only stops the crawler from fetching the page, not from knowing the URL exists. Keeping a page fully out of search results needs a noindex tag instead, which requires the page to be crawlable in the first place so the crawler can actually see the tag. Common Disallow targets include admin areas, internal search result pages, filtered or faceted URLs that create near-duplicate content, and staging environments that shouldn't be found at all.
