Reuters and Time block AI crawlers by default, joining wider industry shift

Reuters and Time have stopped treating AI crawlers as routine traffic and are now admitting them only by permission, a move that gives publishers more leverage over licensing, referrals and the cost of serving machines at scale. The shift marks a break from the old robots.txt model, which depends on bots honoring a site’s instructions rather than asking for access first.

The change, made in May, follows a wider industry turn toward gatekeeping. Reuters says the new setup has not hurt traffic and has lowered the cost of serving bots, a crucial point for newsrooms trying to protect distribution while extracting value from content that AI companies want to train on or surface in search products. Under Reuters’ professional division, bots are approved only when there is a fair-value exchange, which can include licensing, traffic referrals, technical reliability or support for monetization. The Reuters robots.txt file now allows selected crawlers from Amazon, Google, Bing/Microsoft, Yahoo and OpenAI while disallowing others from most of the site.

That posture is spreading fast. Cloudflare said on July 1, 2025, that it would block AI crawlers by default for sites using its service, after more than one million customers had already used its one-click blocking option introduced in September 2024. Cloudflare described the shift as a return to a “permission-based approach” and a “new business model” for the web. Reuters Institute research published in February 2024 found that by the end of 2023, 48% of the most widely used news websites across ten countries were already blocking OpenAI crawlers and 24% were blocking Google’s AI crawler. In the United States, 79% of top news sites were blocking OpenAI at that point.

The pressure has intensified since then. Press Gazette reported on January 22, 2026, that 79% of almost 100 top news websites in the UK and U.S. were blocking at least one AI training crawler, while 71% were blocking AI bots used for retrieval or live search. Tollbit’s Q2 2025 State of the Bots report found that 13.26% of AI bot requests bypassed robots.txt in the second quarter of 2025, up from 3.3% in the fourth quarter of 2024. It also said AI bot and agent traffic surpassed Bingbot activity, while human visitors across sampled sites fell 9.4% from Q1 to Q2 2025.

The message from publishers is becoming harder to miss: access to journalism is no longer free by default. As more outlets move from passive opt-out to active allowlisting, the balance of power shifts toward licensing terms, traffic guarantees and direct bargaining, with consequences that reach well beyond major brands to the economics of digital news itself.