How crawling works
A search bot starts with known URLs, follows links, and retrieves pages. It can also discover addresses through sitemaps, external links, and other sources available to the search engine.
Each request returns a server status and content. The crawler can encounter redirects, crawl restrictions, canonical signals, and resources needed to render the page.
Crawling is not indexing
Retrieving a page is only one stage. The search system still needs to process the material and decide whether it belongs in the index. A page can be crawled repeatedly without appearing in search because it duplicates another URL, lacks useful content, sends conflicting technical signals, or is excluded by the system.
The next stage is covered in the indexing definition.
What can prevent useful crawling
- rules in
robots.txt; - server failures and long redirect chains;
- orphaned pages with no internal links;
- unlimited filter and parameter combinations;
- content that appears only after interactions a crawler does not perform.
Technical SEO keeps important pages accessible while preventing low-value URLs from consuming unnecessary crawl activity.
How crawling is checked
Server logs, search-engine reports, and URL inspection tools can show whether a bot requested a page, which response it received, and whether the primary content was available.