A “Blocked by robots.txt” message means Google could not crawl a URL because a rule in your robots.txt file prevented access. It does not automatically mean the URL cannot appear in Google Search, and it does not always mean you have made a mistake.
The right fix depends on what the page is for. A public article you want people to find needs a different treatment from a filtered search page or a private account area.
This guide helps you identify the rule, choose the right outcome and check the change without accidentally opening up sections you meant to restrict. If you need the basics first, read why robots.txt matters.
Robots.txt and noindex solve different problems
Robots.txt controls crawling. A supported noindex directive tells Google not to include a page in its index. Authentication controls who can actually access content.
| Your goal | Appropriate approach | Important limitation |
|---|---|---|
| Let Google crawl a public page | Remove or narrow the matching Disallow rule | Crawling permission does not guarantee indexing |
| Keep an accessible page out of Google Search | Allow crawling and serve noindex | Google must fetch the page to read the directive |
| Protect private content | Require authentication or another real access restriction | Robots.txt is not a security boundary |
| Reduce crawling of unwanted URL patterns | Use carefully scoped robots.txt rules | A blocked URL can still be discovered and appear in Search |
Google explicitly states that a robots.txt block is not a reliable method for keeping ordinary web pages out of Search. A URL may be known through links even when its content cannot be fetched.
“Blocked” and “indexed, though blocked” are not the same outcome
A blocked URL cannot be crawled under the applicable rules. In some cases Google may still index the URL using information from links elsewhere, without seeing the page content. This explains why you can encounter an “Indexed, though blocked by robots.txt” message or a result without page information.
Do not assume that adding more Disallow rules will remove that result. First decide whether you want the page available in Search or excluded from it.
Also distinguish reports based on an earlier crawl from the site’s current state. A recent edit may not yet be reflected in Search Console.
Step 1: Check the exact URL and crawler
Open the robots.txt file on the same origin as the affected URL. For an illustrative URL such as https://example.com/shop/item, start with https://example.com/robots.txt.
Protocol, hostname and port matter. A file on https://example.com/ does not automatically apply to https://blog.example.com/.
Copy the live file into the Robots.txt Tester, select Googlebot and test the full affected URL. Look at the matching rule rather than guessing from the first Disallow line you see. Test nearby URLs too, including ones that should remain blocked.
Use the syntax reference if you are unsure about groups, path matching, wildcards or end anchors. Paths are case sensitive, so /Shop/ and /shop/ are not interchangeable.
Step 2: Fix pages you want Google to crawl
Suppose this example blocks everything inside a shop:
User-agent: *
Disallow: /shop/
If the entire shop should be crawlable, remove that restriction. If only one public subsection should be accessible, a narrower exception may be more appropriate:
User-agent: *
Disallow: /shop/
Allow: /shop/public/
For Google, the more specific matching path determines the result; an Allow takes precedence when equally specific Allow and Disallow rules conflict. If a separate Googlebot group exists, it can change which rules apply, so do not add an exception to the wildcard group and assume it solves every case.
These are illustrative examples, not instructions to replace your whole file. Keep unrelated restrictions and sitemap declarations intact. The Robots.txt Generator can help draft a revision, but test the output against your real URLs before publishing it.
Step 3: Use noindex when the page should stay out of Search
If visitors should be able to open a page but it should not appear in Google Search, make it crawlable and serve a supported noindex directive. An HTML example is:
<meta name="robots" content="noindex">
An alternative is an HTTP response header:
X-Robots-Tag: noindex
Google does not support Noindex: as a robots.txt directive. Adding that line to robots.txt is not a substitute for a page-level tag or response header.
The common trap is to add noindex while continuing to disallow the page. Google may then be unable to fetch the page and see the instruction. Do not reblock it immediately after adding noindex: the directive needs to remain accessible when Google revisits the URL.
For genuinely confidential pages, use authentication instead. Never rely on a public robots.txt file to hide sensitive paths or data.
Step 4: Check the page response and other barriers
An Allowed result only answers the robots.txt question. It does not establish that the page is healthy or indexable.
Use the HTTP Status Checker to look for error responses, redirect chains and unexpected destinations. Separately inspect the final page’s HTML and response headers for noindex. Check whether a firewall, CDN or login requirement prevents access.
Then use Search Console’s URL Inspection live test for the affected page. For a public page you want indexed, request indexing when appropriate after the live checks pass. For a page you want excluded, focus on making the exclusion directive accessible rather than requesting that it be indexed.
Google decides when to revisit and whether to index a URL. No robots.txt edit guarantees immediate indexing or a ranking improvement.
When you should leave a block alone
Not every blocked URL needs fixing. Internal search results, some faceted URL patterns and other intentionally excluded crawl spaces may be blocked by design.
Before changing a rule, ask:
- Is this a useful public page that should be discoverable in Search?
- Does the rule also protect a large unwanted URL pattern from crawling?
- Would removing it create many duplicate or low-value crawlable URLs?
- Is the real objective exclusion from Search, rather than reduced crawling?
For pages you do want discovered, check that relevant public pages link to them. The Internal Linking Checker can help identify gaps in its sampled crawl; this orphan pages guide explains how to interpret those gaps.
A final verification checklist
- Test the exact URL with the relevant user agent.
- Confirm the intended rule is served in the live robots.txt file.
- Test a page that should remain blocked as well as one that should be allowed.
- Check the final HTTP response and any page-level indexing directives.
- Use a live Search Console inspection and monitor later reports.
The important question is not simply “How do I remove this warning?” It is “Which crawling, indexing and access behavior should this page have?”
Sources
- Google: Introduction to robots.txt
- Google: Block Search indexing with noindex
- Google: Robots.txt specification and rule matching
- Google Search Console: No page information in search results
Reviewed on November 1, 2026.