… The AI Crawler category covers bots that crawl for AI model training, including GPTBot, ClaudeBot, and Bytespider, and is distinct from the AI Assistant and AI Search categories that cover AI Agents. When a request arrives from a verified AI Crawler, Cloudflare reads the response HTML. …
… Using a Content-Signal directive in your robots.txt , you can independently control three things: whether your content can be used for AI training ai-train , whether it can be used as AI input for inference and grounding ai-input , and whether it should appear in search results search : User-agent:… …
… AI Search: the search primitive for your agents AI Search is the search primitive for your agents. Create instances dynamically, upload files, and search across instances with hybrid retrieval and relevance boosting. Just create a search instance, upload, and search. …
… The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals By mid-2025, training drives nearly 80% of AI crawling, while referrals to publishers especially from Google are falling and crawl-to-refer ratios show AI consumes far more than it sends back. …
… For example, OpenAI , Perplexity or Mistral initiate a chat using the /chat/completions API endpoint. Searching through our request logs, we found only a few entries that matched this pattern across our global traffic. …
… Where needed, we can perform quick and easy testing, training, and deployment of ML models. …
… AI Gateway runs in front of our internal AI tools the same way AI Security for Apps runs in front of customer-facing AI features, with the same scoring and the same visibility. …
… This is partly caused by large-scale scraping and model training , and also by user agents acting in ways sites did not anticipate . Websites, therefore, have asked for more technical means to block AI crawlers and associated tools. …
… For example, a plugin that sends an email after a content item gets saved looks like this: import { definePlugin } from "emdash"; export default = definePlugin { id: "notify-on-publish", version: "1.0.0", capabilities: "read:content", "email:send" , hooks: { "content:afterSave": async event, ctx = … …