How to Use

  • Enter rules in the format: User-agent: [bot name]
  • Follow with Disallow: or Allow: directives
  • Separate different user agents with a blank line
  • Use # for comments
  • Click any directive below to add it to the editor

Common Directives

User-agent: *
Applies to all crawlers
Disallow: /
Block entire site
Disallow: /private/
Block a directory
Allow: /public/
Allow a directory
Crawl-delay: 10
Delay between crawls
Sitemap
Link to sitemap
User-agent: Googlebot
Google's crawler
User-agent: Bingbot
Bing's crawler