Loading...
Loading...
Amazon has told publishers that its web crawler, Amazonbot, will begin honoring robots.txt directives starting Monday, June 15, 2026. In an email from Amazon Publisher Support, the company said crawl preferences for Amazonbot will be managed “solely through the industry-standard directives,” replacing prior reliance on manual requests. Site owners who do not add robots.txt rules by the deadline should expect Amazonbot to follow standard web crawling practices. Amazon pointed recipients to docume
Amazon is shifting to industry-standard robots.txt controls for Amazonbot, affecting how sites control crawling. Tech professionals need to update robots.txt or expect Amazonbot to follow default crawl rules starting June 15, 2026.
Dossier last updated: 2026-05-15 00:24:41
Amazonbot is finally respecting robots.txt
Amazonbot is finally respecting robots.txt
Amazonbot is finally respecting robots.txt
Amazon has told publishers that its web crawler, Amazonbot, will begin honoring robots.txt directives starting Monday, June 15, 2026. In an email from Amazon Publisher Support, the company said crawl preferences for Amazonbot will be managed “solely through the industry-standard directives,” replacing prior reliance on manual requests. Site owners who do not add robots.txt rules by the deadline should expect Amazonbot to follow standard web crawling practices. Amazon pointed recipients to documentation at developer.amazon.com/amazonbot and emphasized that robots.txt can control access at the page, directory, or site level and be updated at any time. The change matters for publishers and site operators seeking clearer, self-service control over Amazon’s crawling behavior and server load.