Definition

Crawlability is whether search engine bots can discover and access your pages — blocked by robots.txt, orphan pages, broken links, or login walls.

Detailed Explanation

Improve crawlability: clean site architecture, XML sitemap, internal links, fast server response, no robots.txt blocks on important paths.

Crawlability ≠ indexability — a page can be crawlable but noindexed.

Nepal Context

WordPress sites with broken permalink structures or security plugins blocking bots are common in Nepal. Test with the URL Inspection tool in Search Console and run a live test.

Practical Examples

  • A new service page has no links from the menu or other pages, so Google only finds it weeks later through the sitemap.
  • A security plugin returns a 403 error to Googlebot, so the whole site looks unreachable even though visitors see it fine.
  • Filter combinations on a shop (?color=red&size=m&sort=price) create thousands of URLs that waste crawling on near-duplicates.

How to Check It

  1. In Search Console, open Settings → Crawl stats and look for spikes in server errors or “Host status” problems.
  2. Use URL Inspection → Test live URL on key pages to confirm Googlebot can fetch and render them.
  3. Run a crawler such as Screaming Frog (free up to 500 URLs) and list pages with no internal links pointing to them.

Key Takeaways

  • Bots must reach pages before indexing
  • Internal links are the primary discovery path
  • Fix robots.txt and redirect chains first

Common Mistakes

  1. Orphan pages with zero internal links
  2. Blocking /wp-content/ in robots.txt
  3. Infinite faceted navigation creating crawl traps