The Complete Guide to XML Sitemap

Key takeaways

• An XML sitemap helps crawling and discovery. It does not directly improve rankings or force indexing.
• Only include canonical, indexable, 200-status pages. Redirects, noindex pages, and duplicates don't belong in it.
• Submit the sitemap through Google Search Console and reference it in robots.txt for other crawlers.
• Large sites need a sitemap index file that links to multiple smaller sitemaps, each under the size limits.
• Most sitemap problems are self-inflicted: stale URLs, blocked pages still listed, or a file nobody's updated in months. Set a reminder to spot-check it quarterly.

What an XML sitemap actually is

Structurally, an XML sitemap is a plain text file written in XML that lists URLs, one entry per page, sometimes with extra metadata like the last modified date. Google reads it, cross-references it against what its crawlers already know about your site, and uses it as one input among many for deciding what to crawl and how often.

It’s easy to overstate what a sitemap does. It’s a hint, not a command. Google can and does ignore sitemap entries for pages it considers low quality, duplicate, or not worth indexing, no matter how many times you list them. Think of it as pointing a flashlight at your site’s structure, not as a guarantee anything in the beam gets indexed. That distinction matters because it changes what a sitemap can fix. It speeds up discovery. It cannot manufacture quality a page doesn’t already have.

Why it matters for crawling and indexing

Search engines discover most pages by following links, but link discovery alone is unreliable for new pages, pages buried deep in a site’s structure, or pages with weak internal linking. A sitemap gives a crawler a direct list to check, which shortens the time between publishing a page and a crawler finding out it exists.

The benefit is largest for new sites with little external link equity yet, large sites with thousands of pages where crawl budget is genuinely limited, and sites that just launched or just went through a major restructure. A small, well-linked five-page site gets far less lift from a sitemap than a 10,000-page catalog does, though it’s still worth having either way. Crawl budget itself is a real constraint on big sites specifically. Google allocates a rough crawling allowance per site based on server capacity and perceived value, and a clean sitemap helps that allowance get spent on the pages that matter instead of wasted on dead ends.

What belongs in a sitemap, and what doesn't

Include only canonical URLs that return a 200 status and that you actually want indexed. Every other case is a reason to leave a page out.

• Skip pages marked noindex, since listing them sends a contradictory signal
• Skip redirected URLs (301s and 302s), list the destination instead
• Skip duplicate or parameterized URLs (filtered category views, session IDs), list the canonical version only
• Skip pages blocked by robots.txt, since a crawler can see them in the sitemap but still can’t fetch them

A sitemap padded with pages that shouldn’t be there doesn’t just fail to help. It actively signals to Google that the file isn’t well maintained, which can make it trust the whole thing a little less. A short, accurate list beats a long, messy one every time.

How to submit an XML sitemap to Google

Once the file exists at a stable URL, usually /sitemap.xml, submit it in Google Search Console under Sitemaps, by entering the path and clicking submit. Search Console will report the number of URLs discovered and, over the following days, how many actually got indexed.

Also reference the sitemap’s location in your robots.txt file with a line like Sitemap: https://example.com/sitemap.xml. This helps other crawlers, including Bing and various AI crawlers, find it without needing a separate submission process for each one. Submitting in Search Console is still worth doing on top of that, since it’s the only path that gives you Google’s own coverage and error reporting, including which specific URLs it found but chose not to index and why.

Sitemap index files for large sites

A single XML sitemap is capped at 50,000 URLs and 50MB uncompressed. Once a site crosses that, or approaches it, split the URLs into multiple smaller sitemaps grouped logically, by content type, section, or date, and reference all of them from one sitemap index file that itself lists the individual sitemap URLs.

Grouping by type also makes troubleshooting far easier later. If your product-page sitemap shows an indexing problem but your blog sitemap doesn’t, you already know where to start looking instead of digging through one enormous undifferentiated file.

How a sitemap relates to robots.txt and canonical tags

These three tools do different jobs and people mix them up constantly. Robots.txt tells a crawler where it’s allowed to go. A sitemap tells a crawler where you’d like it to look. A canonical tag tells a crawler which version of a duplicate page is the real one. None of them override each other automatically, and a page can be listed in a sitemap while still being blocked by robots.txt, which is a contradiction worth checking for directly rather than assuming your CMS handles it.

A clean setup has all three agreeing: the canonical URL is what’s in the sitemap, robots.txt allows crawling of that same URL, and nothing noindexed shows up on the list. When one of the three disagrees with the other two, that mismatch is one of the more common reasons pages sit uncrawled for weeks.

Sitemaps for images, video, and news

Beyond the standard page sitemap, Google supports specialized formats for image sitemaps, video sitemaps, and news sitemaps, each with extra fields relevant to that content type (caption and license for images, duration and thumbnail for video). These aren’t required, but they help discovery specifically in Google Images, video search results, and Google News, none of which rely purely on the standard sitemap.

Most sites don’t need all three. A retailer with strong product photography benefits from an image sitemap. A publisher chasing Google News inclusion needs the news format specifically, since standard sitemap entries alone won’t get a site considered for that surface.

Common XML sitemap mistakes

The most common failure is a stale sitemap: a file generated once at launch and never touched again, still listing pages that were deleted or redirected months ago. Close behind it is a sitemap that includes noindexed or blocked pages, which sends Google a mixed signal about whether you actually want those pages found.

A less obvious mistake is treating the sitemap as an SEO fix on its own. It won’t rescue a page that’s thin, duplicate, or genuinely not worth ranking. A crawler finding a bad page faster doesn’t make it a good page. If pages aren’t indexing despite a clean sitemap, the sitemap usually isn’t the problem, the content or the site’s crawl signals are.

Keeping the sitemap current without manual work

On any CMS with more than a handful of pages, generate the sitemap dynamically so it updates automatically as pages are published, unpublished, or redirected, instead of relying on someone to remember to regenerate a static file. Most modern platforms and SEO plugins do this by default, which is one more reason to check that the automated version is actually clean rather than assuming it is.

This work usually sits alongside broader SEO and technical maintenance rather than as a one-time task. A sitemap that was accurate at launch and never checked again slowly drifts out of sync with the live site.

More seo & content guides