Index

An index is the archive of web pages that a search engine can present to users. After a bot crawls and examines a page, the search engine may add it to the index and assess it against various criteria for search results. Webmasters can use robots.txt instructions and the noindex tag to identify pages they do not want indexed.