Indexing

Indexing is the process of adding crawled web pages to a search engine's index. Indexed pages can then be assessed and presented in search results. Before crawling a site, search engine bots examine its robots.txt file and follow the relevant instructions, making that file important to the indexing process.