Crawl budget optimization
Crawl budget is the rate and volume of pages Google crawls on your site. Wasting it on parameter URLs, duplicates, or low-value pages can mean important content is crawled less often. This guide covers how to consolidate signals, use canonicals and robots, and structure your site so crawlers prioritise the right pages.
Identifying waste
Use Search Console and crawlers to see how many URLs exist and which are parameterised, duplicate, or thin. Identify patterns (filters, session IDs, pagination) that add little value. Map which URLs you want indexed and which should be noindexed or disallowed.
- Audit URL count and types
- Find parameter and duplicate patterns
- Define indexable set
Controlling crawl
Use canonical tags to consolidate duplicate URLs. Noindex or disallow low-value paths. Submit a sitemap with only important URLs. Improve internal linking so key pages are close to the homepage. Fix redirect chains and soft 404s.
- Canonicals and noindex
- Clean sitemap
- Strong internal links to key pages