Why is my crawl budget being wasted? on BigCommerce

Crawl budget is the number of pages Google will crawl on your site in a given period.

• Thousands of URL variants from parameters (sort, filter, session) Duplicate or near-duplicate
• Review Search Console Crawl stats and Index Coverage Use a crawler to
• Use canonical tags and parameter handling so parameter URLs don't consume budget

Crawl budget is the number of pages Google will crawl on your site in a given period. When much of it is spent on parameter-heavy URLs, filters, session IDs, thin or duplicate content, or low-priority pages, important content may be crawled less often. Focusing crawl budget on high-value, unique pages helps ensure your key content is discovered and updated in the index. Practical context: prioritise one representative URL, confirm the exact blocker with Search Console and live testing, then apply the fix in templates or settings so the issue does not repeat site-wide. Track impressions, indexed page counts, and click recovery for at least two crawl cycles before closing the task. Practical context: prioritise one representative URL, confirm the exact…

Thousands of URL variants from parameters (sort, filter, session). Duplicate or near-duplicate pages (e.g. pagination, print, share). Low-value or auto-generated pages in the sitemap. Soft 404s or redirects that still get crawled. Internal search or admin URLs exposed. Poor internal linking so crawlers waste time on deep or irrelevant paths.

Common examples

A real-world example: after a site update, a business saw visibility drop for "Crawl budget wasted". They checked Search Console, found the blocking issue, fixed it, and regained impressions over the following crawl cycles.

How to diagnose

Review Search Console Crawl stats and Index Coverage. Use a crawler to see how many URLs exist and which are parameterised or duplicate. Check sitemap: does it include only important URLs? Identify URL patterns that add little value. Look at internal link structure: are key pages easy to reach in few clicks?

Recommended fixes

Use canonical tags and parameter handling so parameter URLs don't consume budget. Remove or noindex low-value pages (filters, session URLs). Submit a sitemap with only indexable, important URLs. Improve internal linking so key pages are linked from homepage or hubs. Fix soft 404s. Use robots.txt to block truly irrelevant paths. Monitor crawl distribution. Practical context: prioritise one representative URL, confirm the exact blocker with Search Console and live testing, then apply the fix in templates or settings so the issue does not repeat site-wide. Track impressions, indexed page counts, and click recovery for at least two crawl cycles before closing the task. Practical context: prioritise one representative URL, confirm the exact blocker with Search Console and live testing, then apply the fix in templates or settings so the issue does not repeat site-wide. Track impressions, indexed page counts, and click recovery for at least two crawl cycles before closing the task. Practical context: prioritise one representative URL, confirm the exact blocker with Search Console and live testing, then apply the fix in templates or settings so the issue does not repeat site-wide. Track impressions, indexed page counts, and click recovery for at least two crawl cycles before closing the task. Practical context: prioritise one representative URL, confirm the exact blocker with Search Console and live testing, then apply the fix in templates or settings so the issue does not repeat site-wide. Track impressions, indexed page counts, and click recovery for at least two crawl cycles before closing the task. Practical context: prioritise one representative URL, confirm the exact blocker with Search Console and live testing, then apply the fix in templates or settings so the issue does not repeat site-wide. Track impressions, indexed page counts, and click recovery for at least two crawl cycles before closing the task.