User-agent: ia_archiver Disallow: / User-agent: *Scrapy* Disallow: / User-agent: *GPTBot* Disallow: / User-agent: * Crawl-delay: 10 Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php