Skip to content

Saint-Barthélemy 56

Escapades et vie locale dans le Morbihan

How a Sitemap Helps Search Engines Discover and Crawl Website Content

An XML sitemap does not cause a URL to be indexed. It notifies crawlers that the URL exists, and they then decide whether or not to crawl it…

Une femme consultant un plan de sitemap imprimé sur un bureau en bois, illustrant la structure et la navigation d'un site web
5 minutes

An XML sitemap does not cause a URL to be indexed. It tells crawlers that the URL exists; they then decide whether to crawl it and, ultimately, include it in their index. Confusing these three stages (discovery, crawling, indexing) leads to overestimating the file’s importance or, worse, overlooking real crawl issues. Understanding how this works lets you use a sitemap as a diagnostic tool, not as a magic SEO lever.

Orphan URLs and sitemaps: measuring what crawlers can’t find on their own

An orphan URL belongs to a page that is missing from the internal link structure, with no links connecting it to the rest of the site. Without a sitemap, crawlers never find it. With a sitemap, it appears in the discovery queue, but there’s no guarantee it will be crawled or indexed.

You can verify the distinction in Google Search Console, in the “Page indexing” report. A URL listed in the sitemap may remain in the “Discovered – currently not indexed” status for weeks. This status means Googlebot knows the URL but hasn’t yet deemed it a priority. Discovery does not mean indexing, and the gap between the two reveals how Google perceives the page’s quality or how much crawl budget remains.

To identify genuinely orphaned URLs, we cross-reference three sources: the sitemap URL list, server crawl logs, and the indexing report. A URL that appears in the sitemap but is absent from the logs over a thirty-day period has not been visited. If it also appears as “Not indexed,” the issue goes beyond the sitemap: the internal linking or the content itself is in question.

A well-filtered sitemap therefore serves as a reference inventory. It makes visible pages that the site itself doesn’t highlight—something no conventional link-based crawl can do.

A web developer analyzing a hierarchical sitemap displayed on a large wall-mounted screen in a modern tech office

lastmod, changefreq, and priority tags: what Google actually uses in an XML sitemap

The sitemap protocol defines three optional metadata fields for each URL. In practice, Google ignores the changefreq and priority tags. Keeping them in the file won’t cause an error, but they don’t affect crawl frequency or the relative ranking of pages.

The lastmod tag remains useful under one strict condition: the date must reflect a genuine, substantial change to the content. Updating this value every time the sitemap is generated automatically, without changing the page, dilutes the signal. Googlebot eventually considers the date unreliable and stops using it to schedule revisits.

By comparison, La Top Liste’s sitemap illustrates a well-structured file in which every listed URL points to an active page, without padding it with obsolete addresses. This kind of clean sitemap makes the crawler’s job easier because useful URLs aren’t buried in noise.

Clean up the file instead of bloating it

A sitemap containing redirected URLs (301s), pages returning a 404 error, or URLs blocked by robots.txt sends conflicting signals. The crawler discovers the URL, tries to crawl it, then rejects it. This cycle uses crawl budget without producing any results.

  • Exclude any URL that returns an HTTP status code other than 200 when the file is generated.
  • Remove pages with a noindex tag, since listing a URL in the sitemap while asking search engines not to index it creates a technical contradiction.
  • Limit the file to canonical URLs: if two addresses point to the same content via rel="canonical", only the canonical version belongs in the sitemap.

A sitemap limited to URLs that can actually be indexed gives an accurate picture of the content the site wants to make available to search engines.

Sitemaps and language versions: declaring hreflang equivalents

On a multilingual site, an XML sitemap can use hreflang annotations to link language versions of the same page. Each <url> entry can contain multiple xhtml:link elements specifying the language and corresponding URL.

This method complements the HTML <link rel="alternate" hreflang="x"> tags placed in the <head>. For sites with several hundred translated pages, centralizing these declarations in the sitemap makes maintenance easier. For example, a change to the Spanish version’s URL can be made in a single file instead of updating every source page.

Google accepts all three methods (HTML tags, HTTP headers, and sitemaps) and treats them as equivalent. A sitemap doesn’t replace the other methods, but it provides a single point of control—useful when the CMS doesn’t natively manage hreflang tags in page code.

Hands annotating a sitemap wireframe on a tablet in a café, alongside a notebook and a cup of coffee

Crawl budget and sitemaps: when the file actually changes crawling

For a site with a few dozen well-linked pages, a sitemap offers only a marginal benefit. Crawlers follow internal links and discover all the content in just a few passes.

A sitemap becomes essential when a site has several thousand URLs or when its structure creates areas with few internal links. Typical examples include:

  • E-commerce sites with product pages accessible only through faceted filters, making them invisible from the main navigation.
  • News sites publishing dozens of articles a day, where content quickly disappears from the homepage.
  • Isolated silo architectures, where some sections receive no links from other sections.

In these situations, the sitemap acts as a safety net. It doesn’t replace a coherent internal linking structure, but it compensates for structural gaps until they can be fixed.

Submit and monitor

Adding the sitemap to the robots.txt file (using the Sitemap: directive) and submitting it through Google Search Console or Bing Webmaster Tools lets you monitor how it’s processed. The Search Console report shows the number of URLs submitted, the number indexed, and any errors encountered while reading the file.

The gap between submitted URLs and indexed URLs is the first metric to monitor. A low ratio points either to a content quality issue or to a sitemap cluttered with URLs that can’t be indexed. In either case, the fix lies in the site itself, not the file.

A sitemap remains a declaration and diagnostic tool. It makes clear what the site offers search engines, but never replaces their independent indexing decisions.

How a Sitemap Helps Search Engines Discover and Crawl Website Content