Config Files · 1 file
Sample robots.txt File
robots.txt tells web crawlers which paths they may fetch. Our sample is realistic and valid for its tool, with fake hosts and secrets.
robots.txt · 241 B
robots.txt
A robots.txt with per-crawler groups, Allow/Disallow rules, wildcards, a crawl delay and a Sitemap line. All hosts, keys and secrets are fake.
CC0 · free for any use · Verified by an independent tool · Report a problem
# Sample robots.txt — samplefiles.com
User-agent: *
Disallow: /admin/
Disallow: /search?
Disallow: /*.pdf$
Allow: /admin/public/
User-agent: GPTBot
Disallow: /
User-agent: Bingbot
Crawl-delay: 5
Sitemap: https://example.com/sitemap.xmlFile details
Download from the command line
curl -L -o robots.txt https://files.samplefiles.com/config/robots-txt/robots.txtwget https://files.samplefiles.com/config/robots-txt/robots.txthttps://files.samplefiles.com/config/robots-txt/robots.txtprintf '%s %s\n' c72077d6da140d1a3532567836e8076d6cb8a9ae3517f0fbd54cdac75696df3f robots.txt | shasum -a 256 -cOr check the download in your browser with the checksum calculator.