robots.txt and sitemap.xml: different jobs, common confusion
Understand what robots.txt and XML sitemaps do, what they do not do and which launch mistakes to avoid.
robots.txt and sitemap.xml are often discussed together, but they solve different problems. robots.txt provides crawl directives for compliant crawlers. A sitemap provides a structured list of URLs you want search systems to discover.
robots.txt is not an indexing guarantee
Blocking a URL in robots.txt prevents compliant crawlers from fetching it, but it is not the same as a noindex directive.
A sitemap is not a ranking shortcut
A sitemap helps discovery. It does not make thin, duplicate or low-value pages rank automatically.
Watch the production configuration
Development sites often carry restrictive robots rules or noindex tags. Compare production settings to what you actually want crawlers to access.
A simple robots.txt can be enough
Many small sites need only a minimal robots file. Complexity should come from a real crawling requirement, not from copying directives from a much larger website. Every additional rule is another opportunity to block something unintentionally.
Keep noindex in the page or response
When a page should not appear in search, a noindex directive belongs where the crawler can read it, such as a robots meta tag or supported HTTP response header. Blocking the URL in robots.txt can prevent a crawler from fetching the page and therefore from seeing a page-level directive.
What belongs in a sitemap
Prefer canonical, indexable URLs that you actually want discovered. Avoid filling a sitemap with redirects, 404 pages, duplicate parameter URLs or internal utility endpoints. A sitemap is more useful when it represents the site's intended public structure.
Check both files after migrations
Domain changes and CMS migrations can leave old hostnames in sitemap entries or carry development robots rules into production. After any migration, open both files directly and search for obsolete domains, staging paths and unexpected disallow rules.
Useful troubleshooting questions
- Does the file return HTTP 200?
- Is it served from the hostname you actually use?
- Does robots.txt reference the correct sitemap if you include a Sitemap directive?
- Are sitemap URLs canonical and HTTPS?
- Did a deployment accidentally add noindex to templates?