A robots.txt file contains access instructions for search engine bots. Crawlers typically examine it before exploring a website and use its rules to determine which pages they may crawl. Because these instructions shape bot access, webmasters need to take particular care when creating or editing the file.
