# DocHuntr robots.txt User-agent: * Allow: / # Sitemap Sitemap: https://dochuntr.net/sitemap.xml # Crawl-delay (respectful) Crawl-delay: 10