Cloudflare introduces AI Audit, a free tool for websites to analyse and monitor how artificial intelligence models (AI models) are using their content. With it, website makers and content creators can now quickly and easily see how AI model providers are using their content and determine whether and how the models can access it.
Organisations may be barely aware that AI bots are scanning their website content thousands of times a day, without the content creator knowing or receiving compensation for it. For companies large and small, this leads to significant loss of value. Even if website owners know how AI bots are using their content, they do not have sophisticated tools to determine which scanning activities they are allowing and easily take action against it. If society as a whole is to continue to benefit from the rich diversity of internet content, content creators need the right tools to get a grip on the situation.
With AI Audit, Cloudflare is going to offer insight and control to content creators. This will allow a transparent exchange between websites that want more control over their content and AI model providers that need new data sources. Everyone benefits. Through this solution, Cloudflare would like to help all website owners with the following:
* Free automated audit of AI bots: AI is evolving at lightning speed. Many website owners need time to understand and analyse how AI bots affect their traffic or business. Many small sites do not have the skill or capacity to block AI bots manually. But if content creators can block all AI bots at the click of a button, they regain control.
* Using statistics to see how AI bots are consulting their content: all websites using Cloudflare now have access to statistics to understand why, when and how often AI models are consulting their website. Website owners can now distinguish between different bots, such as text generation bots that cite the source of the data when generating a response, versus bots that import data without citing or referencing the source.
* Better protect rights during negotiations with model providers: a growing number of sites are signing agreements directly with model providers to give permission for training and content search in exchange for payment. Cloudflare's AI Audit tab provides advanced metrics to understand what data is often used in these negotiations, such as the crawl speed of the entire page or certain parts of it. Cloudflare also provides models for terms of service that content creators can add to their website to legally protect their rights.
* Setting a fair price for content scanning rights and seamless payments (in development): many website owners, whether they are the large companies of the future or individual bloggers, do not have the resources, context or expertise to make one-off deals that larger publishers agree with AI model providers. Moreover, AI model providers do not have the capacity to do so with every website that approaches them. In the future, even the biggest content creators will benefit from Cloudflare's seamless process for pricing and payments. This will make it easier for model providers to find new, scannable content they wouldn't otherwise have access to. In turn, content providers regain control while being paid for the value they create.