Is my robots.txt blocking my site? on Shopify

The robots.txt file tells search engine crawlers which paths they may or may not request.

• A global Disallow / blocks all crawlers from the entire site Disallow
• View your live robots txt at yourdomain com/robots txt and look for
• Remove or narrow any Disallow rule that blocks content you want indexed

The robots.txt file tells search engine crawlers which paths they may or may not request. A misconfigured robots.txt—for example, Disallow: /—can block all or most of your site from being crawled, so pages never get indexed. Even partial blocks can hide important sections. Checking what's actually blocked and fixing overly broad rules restores crawl access so your content can be discovered and ranked. Practical context: prioritise one representative URL, confirm the exact blocker with Search Console and live testing, then apply the fix in templates or settings so the issue does not repeat site-wide. Track impressions, indexed page counts, and click recovery for at least two crawl cycles before closing the task. Practical context: prioritise one representative URL, confirm the…

A global Disallow: / blocks all crawlers from the entire site. Disallow rules for admin or staging paths may be too broad and accidentally block key areas. Some platforms or plugins add aggressive rules by default. Copy-paste errors or misunderstandings about syntax can block more than intended. Legacy rules from an old site or migration might still be in place. Verifying the live robots.txt and testing with Search Console or a tester tool reveals the impact.

Common examples

A real-world example: after a site update, a business saw visibility drop for "Robots.txt blocking site". They checked Search Console, found the blocking issue, fixed it, and regained impressions over the following crawl cycles.

How to diagnose

View your live robots.txt at yourdomain.com/robots.txt and look for Disallow rules. Use Google Search Console's robots.txt Tester (under Settings) to see how Google interprets the file and which URLs are blocked. Use URL Inspection on important URLs to see if they're blocked by robots.txt. Check whether a CDN, plugin, or server config is serving a different robots.txt than you expect.

Recommended fixes

Remove or narrow any Disallow rule that blocks content you want indexed. Allow Googlebot (and Bot if you use it) for paths that should be crawlable. Use specific paths (e.g. /admin/, /cart/) rather than blocking the whole site. Submit the updated robots.txt and use Search Console to confirm Google can crawl key URLs. After fixing, request indexing for important pages via URL Inspection and monitor Coverage for improvements. Practical context: prioritise one representative URL, confirm the exact blocker with Search Console and live testing, then apply the fix in templates or settings so the issue does not repeat site-wide. Track impressions, indexed page counts, and click recovery for at least two crawl cycles before closing the task. Practical context: prioritise one representative URL, confirm the exact blocker with Search Console and live testing, then apply the fix in templates or settings so the issue does not repeat site-wide. Track impressions, indexed page counts, and click recovery for at least two crawl cycles before closing the task. Practical context: prioritise one representative URL, confirm the exact blocker with Search Console and live testing, then apply the fix in templates or settings so the issue does not repeat site-wide. Track impressions, indexed page counts, and click recovery for at least two crawl cycles before closing the task.