Amazonbot to align with robots.txt standards from June 2026

Amazon has announced that its web crawler, Amazonbot, will begin adhering to industry-standard robots.txt directives starting Monday, 15 June 2026. The company stated that this change provides website owners with direct control over how the bot accesses their sites, replacing the need for manual requests. The announcement was communicated via email, which the recipient noted contained technical headers suggesting it was sent from Outlook for Mac. The developer of the Anubis project indicated they would integrate these changes into their own systems.

The notification, distributed directly to website owners, outlines that crawl preferences will be managed solely through these standard directives from the effective date. Amazon clarified that if no directives are implemented by the deadline, Amazonbot will follow standard web crawling practices. The email provided instructions on maintaining current preferences via page-, directory-, or site-level directives, allowing updates at any time.

Technical observations from the recipient highlighted discrepancies in the email's metadata. While the message retained a "sent from my iPhone" signature, the inclusion of Exchange-specific headers suggested the communication originated from Outlook for Mac. These details were noted in the verbatim copy of the message shared by the recipient.

The Anubis project, a tool developed in response to Amazon's previous data scraping methods, is set to incorporate these changes. The developer of the project indicated an intention to merge the robots.txt updates into their system, acknowledging the shift in Amazon's operational approach.

This development marks a significant adjustment in how Amazon manages its web crawling activities. By aligning with established industry standards, the company aims to provide clearer mechanisms for site owners to regulate bot access, potentially reducing the reliance on ad-hoc manual requests that characterised previous interactions.

The announcement has been met with mixed reactions within the technical community. While some view the move as a positive step towards standardisation, others note the historical context of Amazon's scraping practices, which contributed to the creation of tools like Anubis. The integration of these changes into third-party systems remains a point of interest for developers monitoring web data access.

As the June 2026 deadline approaches, website owners are advised to review their robots.txt configurations to ensure compliance with the new directives. The shift underscores the growing importance of transparent and standardised protocols in managing web crawler interactions across the digital landscape.