Indexing


Indexing is what happens after crawling: once a search engine has fetched and read a page, it decides whether to store that page in its index, the enormous database of pages a search engine actually draws results from. A page has to be indexed to have any chance of appearing in search results at all; ranking is a separate question that only applies to pages that clear this bar first.

Being crawled doesn't guarantee indexing. A search engine can crawl a page and still leave it out, because the content is judged too thin or duplicative, a noindex tag says not to, or the page simply isn't considered valuable enough relative to everything else competing for index space. Google Search Console's URL Inspection tool reports a page's indexing status directly, along with a reason when a page has been excluded, which makes it the first place to check when a page isn't showing up in search results the way it should be. robots.txt affects crawling specifically, not indexing directly, though a page that's never crawled obviously can't be indexed either, which is why the two get discussed together even though they're distinct steps.

Indexing
Explore tailored SEO consulting, technical audits, and performance strategies designed for ambitious teams.
Talk to an SEO expert

Contact us

How we handle your data

Open as a full page