Definition

Robots.txt is a text file at your site root (/robots.txt) that tells search engine crawlers which URLs they may or may not request — a crawl directive, not an indexation guarantee.

Detailed Explanation

User-agent: *
Disallow: /admin/
Disallow: /search/
Allow: /blog/

Sitemap: https://example.com/sitemap.xml

Disallow prevents crawling; it does not always prevent indexing if URLs are linked externally. Use noindex meta tag to prevent indexing.

Never block CSS, JS, or image paths needed for rendering.

Nepal Context

Nepali WordPress sites often accidentally Disallow /wp-content/ breaking rendering in Search Console. Audit robots.txt after every migration or SEO plugin install.

Practical Examples

  • An e-commerce site disallows /cart/ and /checkout/, which have no search value.
  • A site blocks internal search URLs (/search?q=) to stop endless low-value pages being crawled.
  • A developer leaves Disallow: / from staging on the live site, and organic traffic stops within weeks.

How to Check It

  1. Open yourdomain.com/robots.txt in a browser and read every Disallow line.
  2. In Search Console, Settings → robots.txt shows the version Google last fetched and any errors.
  3. Use URL Inspection on important pages; it reports “Blocked by robots.txt” if a rule catches them.

Key Takeaways

  • robots.txt = crawl control, not index control
  • Always include Sitemap directive
  • Check the robots.txt report in Search Console (Settings → robots.txt)

Common Mistakes

  1. Blocking entire site during development and forgetting to unblock
  2. Using robots.txt to hide pages you want deindexed (use noindex instead)
  3. Blocking resources Google needs to render JavaScript pages