Answer

What Is Crawl Budget?

Crawl budget is the number of URLs Google is willing and able to crawl on your site. For most small sites it is not the problem. For large or messy ones it can be.

Crawl budget is the number of pages Google will crawl on your site within a set period. It is the combination of how much crawling your server can handle and how much Google wants to crawl. If Google runs out of budget before reaching your important pages, they get found late or not at all.

What crawl budget is made of

Google describes two parts. The first is the crawl capacity limit: how fast Google can fetch your pages without overloading your server. A slow or error prone server lowers it. The second is crawl demand: how much Google wants to crawl, based on how popular your URLs are and how often they change.

Put together, those two decide how many of your URLs get crawled and how often.

Who it matters for

Google's own guidance says crawl budget is a concern mainly for very large sites, with a million or more unique pages that change about weekly, or sites with around ten thousand pages that change daily. Sites with a lot of pages marked as discovered but not crawled in Search Console are also worth checking.

If you run a normal business site with a few hundred pages, an unindexed page is rarely a budget problem. Look at quality, duplication and technical blocks first. Our answer on why pages are not indexed covers those.

What wastes crawl budget

Crawling gets wasted on URLs that add no value. The usual culprits are faceted navigation and filter URLs, endless parameter combinations, session IDs, duplicate pages, long redirect chains and soft 404 errors. Each one pulls Google's attention away from pages you want ranked.

How to check yours

In Search Console, open Settings, then the Crawl Stats report. It shows total crawl requests, response times and the types of response Google received. Rising errors or slow responses lower your crawl capacity. A large share of requests going to junk URLs shows where the waste is.

Your server logs give a sharper picture, because they show exactly which URLs Googlebot requested.

Where to go next

Fixing crawl waste is part of our technical SEO fixes work, and our full crawl budget guide walks through the checks and fixes step by step.

FAQ

Questions owners ask us

Do small sites need to worry about crawl budget?

Usually no. Google says crawl budget matters mainly for very large sites, or sites with many pages that change often. If a small site has pages not indexed, the cause is more often quality, duplication or a technical block than budget.

Does blocking pages in robots.txt save crawl budget?

It stops Google from crawling those URLs, but Google says blocking pages does not move their budget to other pages unless your server is at its limit. Blocked pages can also still appear in results if they are linked, and Google cannot see a noindex tag on a page it is not allowed to crawl. Use it carefully.

Get started

Tell us where it hurts.

Your site, what changed and when. We reply with a first read on the cause.