Definition

Crawl budget is the number of pages search engine bots will crawl on your site within a given timeframe — determined by crawl capacity (server health) and crawl demand (URL importance and freshness).

Detailed Explanation

Google allocates limited crawl resources per site. Small sites (<1,000 pages) rarely hit limits. Large sites, e-commerce catalogs, and sites with many low-value URLs (filters, tags, pagination) must actively manage crawl budget.

Improve efficiency: fix 404s/500s, eliminate duplicate URLs via canonicals, block low-value paths in robots.txt, improve page speed, and use internal linking to prioritize important pages.

Nepal Context

Nepali news portals and large e-commerce catalogs (faceted filters, city pages) are most at risk. Block internal search results, thin tag pages, and parameterized URLs from crawling. Prioritize product, category, and service money pages.

Practical Examples

  1. Audit “Discovered - currently not indexed” in Search Console — a large, growing count can mean Google is not getting round to crawling those URLs.
  2. robots.txt Disallow: /search/, /*?sort=, /tag/ if thin.
  3. Log file analysis: which URLs consume most crawl slots?

How to Check It

  1. In Search Console, open Settings → Crawl stats and review total requests, average response time, and responses by type.
  2. Compare crawled URL patterns against the pages you want indexed; many requests to filters, tags, or parameters show waste.
  3. If you have server access, check access logs for Googlebot requests to see exactly which URLs it spends time on.

Key Takeaways

  • Crawl budget matters for large/thin-content-heavy sites.
  • Server errors and slow TTFB reduce crawl capacity.
  • Quality internal links increase crawl demand for key pages.

Common Mistakes

  1. Blocking CSS/JS in robots.txt — hurts rendering and indexing.
  2. Infinite faceted navigation — millions of crawlable filter combinations.
  3. Ignoring Search Console crawl stats — miss server capacity issues.