Cloudflare announced that a setting to block AI web crawlers would be enabled by default for its customers. This announcement comes almost two years after the setting was originally introduced to protect content creators and prioritize important traffic.

At first, the setting only worked for web crawlers respecting robots.txt, which many AI web crawlers are starting to ignore (see below). Some AI web crawlers also mask themselves as being from search engines. Because of these issues, Cloudflare later updated the feature to apply to any detected AI web crawler and allowed a total ban on them.