Demystifying evals for AI agents
…Through our internal work and with customers at the frontier of agent development, we’ve learned how to design more rigorous and useful evals for agents. Here's what's worked across…
Search
The Long-Term Benefit Trust
Announcements The Long-Term Benefit Trust Sep 19, 2023 Today we are sharing more details about our new governance structure called the Long-Term Benefit Trust (LTBT) , which we have been developing…
Quantifying infrastructure noise in agentic coding evals
…passive container, but an integral component of the problem-solving process. Two agents with different resource budgets and time limits aren't taking the same test. Eval developers have begun accounting for…
A global workspace in language models
…We find that Claude has developed a small collection of internal neural patterns that, compared to all its other internal processing, play a special role. We call the collection of these patterns…
Investigating three real-world incidents in our cybersecurity evaluations
…These evaluations are a critical step in developing and releasing models—a nuanced understanding of model capabilities allows us to design appropriate safeguards and ultimately release models safely. We began running cybersecurity…
Agents for financial services
…financial models from filings, data feeds, and analyst inputs; Market researcher tracks sector and issuer developments, synthesizes news, filings, and broker research, and flags items for credit and risk review. Finance and…
Anthropic Economic Index report: Economic primitives
…software development ("Help debug, develop, and optimize software across multiple programming domains") and personal life management ("Assist with personal life management and everyday tasks"). Figure 2.2 shows the primitive profile for…
Patterns and problems in multiagent systems
…they typically develop rich—and dynamic—interdependencies as they evolve. To test how well swarms of agents could coordinate on a project like this, we directed several swarms to each create a…
Core views on AI safety: When, why, what, and how
…One particularly important dimension of uncertainty is how difficult it will be to develop advanced AI systems that are broadly safe and pose little risk to humans. Developing such systems could lie…
Anthropic Economic Index report: Learning curves
…We sample 1 million conversations from both Claude.ai, our consumer-facing web product, and our first-party API, the developer-facing interface for integrating Claude into products and workflows. 2 Coding…