Crawling is the process a search engine bot uses to discover and read through a website's pages, following links from one page to the next the way a person browsing the site might, except systematically and at far greater scale. Googlebot is the best-known example, but any automated tool that fetches and reads pages this way, an SEO auditing tool, an uptime monitor, is crawling in the same technical sense.
A crawl doesn't just note that a page exists; it reads the content, images, and outbound links on it, which is how new pages and new links get discovered in the first place. Crawling and indexing are related but distinct steps: a page has to be crawled before it can be indexed, but being crawled doesn't guarantee indexing, since a search engine can still decide a page isn't worth adding to its index. robots.txt and meta robots tags are the main ways a site controls what gets crawled at all, separate from the noindex directive that controls what gets indexed once a page has already been seen.
