What happens during indexing
After crawling, a search system processes the page's main text, headings, images, links, language, and technical signals. It determines whether the URL represents a distinct resource and which version should be treated as canonical.
If the page is eligible, the system stores information about it in the index. This allows the engine to match stored documents to queries without recrawling the entire web for every search.
How indexing differs from crawling
Crawling means a bot requested the URL and retrieved content. Indexing means the search system processed the page and chose to retain it as a possible result. The first event does not guarantee the second.
Why a page may not be indexed
- the page contains a
noindexdirective; - the system selects another canonical URL;
- the content substantially duplicates a known page;
- the server returns an error or unstable response;
- the page contains little distinct information;
- the search system has not processed recent changes yet.
Meeting the technical requirements does not obligate a search engine to index a URL.
How indexing status is checked
For Google, site owners can use URL Inspection and indexing reports in Search Console. A diagnosis may also review the server response, robots directives, canonical URL, internal links, and the availability of the main text.