freeseobot

Noindex vs robots.txt: How to Keep a Page Out of Google

Robots.txt controls crawling, not indexing. Learn when to use noindex, the X-Robots-Tag header or a password, and the mistake that keeps pages in results.

By freeseobot Editorial · Published · 5 min read

To keep a page out of Google, use noindex, not robots.txt. Google's own documentation says robots.txt "is not a mechanism for keeping a web page out of Google" and that to keep a page out, you should "block indexing with noindex." The two tools do different jobs, and mixing them up is one of the most common ways a page that should be hidden ends up in search results.

This guide explains the difference, shows the exact code for each option, and lists the mistakes to check for on your own site.

What is the difference between crawling and indexing?

Crawling is a bot fetching a URL. Indexing is a search engine storing what it found so the page can appear in results. They are separate steps, and each has its own control:

You want to... Use Controls
Stop bots requesting a URL robots.txt Disallow Crawling
Keep a page out of search results noindex Indexing
Keep a page private from everyone Password or login Access

Google's robots.txt introduction says the file is used mainly to avoid overloading your site with requests. It tells crawlers which URLs they may access. It does not tell Google what to show.

Why does a robots.txt block not remove a page from Google?

Because a blocked URL can still be indexed from other sources. Google's documentation says a disallowed URL can still appear in results if other sites link to it, and the result may show without a description. Google cannot read the page, so it shows what it knows from the links pointing at it.

Google's page on robots.txt also warns: "Don't use a robots.txt file as a means to hide your web pages."

How do I add noindex to a page?

You have two options. Google says both have the same effect.

Option 1: a meta tag in the page <head>.

<meta name="robots" content="noindex">

To target only Google's crawlers, use googlebot as the name:

<meta name="googlebot" content="noindex">

Option 2: an HTTP response header. This is the only option for files that have no HTML, such as PDFs, images and videos.

HTTP/1.1 200 OK
X-Robots-Tag: noindex

Any rule you can write in the meta tag can also go in the header. Server configuration can apply the header to a whole folder or file type at once. The rule values and the name and content attributes are not case-sensitive.

Google does not support noindex inside robots.txt, so do not put it there.

What is the most common noindex mistake?

Blocking the page in robots.txt and adding noindex to the same page. This does not work. Google's documentation states that the page must not be blocked by robots.txt. If it is, the crawler never sees the noindex rule, and the page can still appear in results when other pages link to it. The robots meta tag guide says the same: rules can be read and followed only if crawlers are allowed to access the page.

The fix is simple:

  1. Remove the Disallow line for that URL from robots.txt.
  2. Keep the noindex tag or header on the page.
  3. Wait for Google to crawl the page again.

Once Google has dropped the page, you can decide whether to block crawling too. For most small sites there is no reason to.

How long does noindex take to work?

Google gives no fixed timeline. Its documentation says that, depending on the importance of the page on the internet, "it may take months for Googlebot to revisit a page." Until then the page may still appear.

You can ask for a recrawl with the URL Inspection tool in Search Console. If you need a page gone faster, Google points to its separate removals documentation. The Page Indexing report in Search Console lists pages where Googlebot found a noindex rule, so you can confirm your tag was seen.

What if noindex and other rules conflict?

The more restrictive rule wins. Google's example: if max-snippet:50 and nosnippet both apply to a page, nosnippet takes effect. When several crawlers are targeted with different rules, Google uses the sum of the negative rules.

Related rules you may see next to noindex:

  • nofollow: do not follow the links on the page.
  • nosnippet: show no text snippet or video preview. A static image thumbnail may still appear.
  • max-snippet:[number]: limit the text snippet to that many characters. 0 is the same as nosnippet.

Which pages should you noindex?

Use noindex for pages that are useful to visitors but pointless in search results. Typical examples:

  • Thank-you and order confirmation pages.
  • Login, account and cart pages.
  • Internal search results pages.
  • Staging copies and old campaign pages.

Do not use noindex to choose between two near-identical pages. Google's page on consolidating duplicate URLs advises against it for picking a canonical within one site. Use a redirect or a canonical tag instead. Our guide to canonical tags covers that.

Should I put noindex pages in my sitemap?

No. A sitemap is a list of pages you want indexed, so a noindex URL in it sends a mixed message. The XML sitemap guide explains what to include and what to leave out.

How do I find stray noindex tags on my own site?

The common accident is a noindex left over from development, or a plugin setting that applies it to a whole section. Check for it:

  1. Run the freeseobot SEO checker. It lists pages set to noindex and flags a noindex on the home page or on a page listed in your sitemap.
  2. Open the page source of your most important pages and search for noindex.
  3. Check the HTTP headers for X-Robots-Tag, since a header is invisible in the page source.
  4. In Search Console, open the Page Indexing report and look for "Excluded by 'noindex' tag".

If a page you want in Google is missing, read why isn't my page indexed for every status.

Questions

Can I use robots.txt to hide a private page?

No. Anyone can read robots.txt, so listing a private URL there advertises it. Google recommends noindex or password protection. For truly private content, use a login.

Does noindex stop other search engines too?

<meta name="robots" content="noindex"> addresses all supporting search engines. Google documents it for its own crawlers, and we have not tested every other engine, so check the documentation for each one you care about.

Will noindex hurt the rest of my site?

Google's documentation we read does not describe any penalty for the rest of the site. It only removes the tagged page from results.

Should I block AI crawlers with noindex?

No. Noindex is a search results rule. Controlling AI crawlers is done in robots.txt by user agent. See robots.txt for AI crawlers.

I removed noindex but the page is still missing. Why?

Google must crawl the page again to see the change. Request indexing for it in URL Inspection and give it time.

Sources

More guides