Introducing Claude Opus 4.7
…1 Real-world work . As well as its state-of-the-art score on the Finance Agent evaluation (see table above), our internal testing showed Opus 4.7 to be a more…
Search
People also ask
Why build evaluations?
When teams first start building agents, they can get surprisingly far through a combination of manual testing, dogfooding, and intuition. More rigorous evaluation may even seem like overhead that slows down shipping. But after the early prototyping stages, once an agent is in production and has started scaling, building without evals starts to break down. The breaking point often comes when users report the agent feels worse after changes, and the team is “flying blind” with no way to verify except to guess and check. Absent evals, debugging is reactive: wait for complaints, reproduce manually
Demystifying evals for AI agentsIs an LLM’s knowledge useful in an applied scenario?
In considering the contribution of AI to biorisk, we need to know more than just how well it performs on a quiz. We need to look at evaluations that involve real people, and closely mirror our actual threat scenarios. Moreover, just as we benchmark AI knowledge by comparing it to experts, we need to measure AI utility by comparing it to the most easily accessible alternative—in this case, the internet. To meet both of these criteria, we have conducted several controlled trials measuring AI’s ability to assist in the planning of a hypothetical bioweapons acquisition process. Participants were g
LLMs and bioriskHarness design for long-running application development
…But for the parts of the build that were still at the edge of the generator’s capabilities, the evaluator continued to give real lift. The practical implication is that the evaluator…
Natural Language Autoencoders
…These high-stakes tests are simulations, not real-world scenarios. Nevertheless, we would like to use them to understand how Claude would behave if they were real. But there’s a hitch…
Measuring LLMs’ ability to develop exploits
…large fraction of real-world harm comes from N-days: vulnerabilities that have already been publicly disclosed, but only patched on some devices. In this post, we evaluate how much large language…
Experimenting with AI to defend critical infrastructure
…large fraction of real-world harm comes from N-days: vulnerabilities that have already been publicly disclosed, but only patched on some devices. In this post, we evaluate how much large language…
Equipping agents for the real world with Agent Skills
Engineering at Anthropic Equipping agents for the real world with Agent Skills Update: We've published Agent Skills as an open standard for cross-platform portability. (December 18, 2025) As model capabilities…
Finding bugs with Claude and property-based testing
…large fraction of real-world harm comes from N-days: vulnerabilities that have already been publicly disclosed, but only patched on some devices. In this post, we evaluate how much large language…
Introducing Claude Opus 4.5
…Claude Opus 4.5 is state-of-the-art on tests of real-world software engineering: Opus 4.5 is available today on our apps, our API, and on all three major…
Eval awareness in Claude Opus 4.6’s BrowseComp performance
…accomplish a task, and how difficult it will be to constrain its behavior in the real world, particularly on complex, compute-intensive, long-running tasks, which increase the likelihood of an agent…
Measuring LLMs' impact on N-day exploits
In cybersecurity, a large fraction of real-world harm comes from N-days: vulnerabilities that have already been publicly disclosed, but only patched on some devices. In this post, we evaluate how…