Demystifying evals for AI agents
…We’ve found this approach too rigid and results in overly brittle tests, as agents regularly find valid approaches that eval designers didn’t anticipate. So as not to unnecessarily punish creativity…
Search
Project Vend: Can Claude run a small shop? (And why does that matter?)
…Another employee suggested Claudius start relying on pre-orders of specialized items instead of simply responding to requests for what to stock, leading Claudius to send a message to Anthropic employees in…
Project Vend: Phase two
…But the capabilities of large language models in areas like reasoning, writing, coding, and much else besides are increasing at a breathless pace. Has Claudius’s “running a shop” capability shown the…
2028: Two scenarios for global AI leadership
…America and its allies approach AI competition from a position of great strength. The tools for AI dominance have been built by an exceptionally innovative ecosystem of companies in democratic nations. Our…
Vibe physics: The AI grad student
…Even as these approaches are visionary, their successes to date seem a bit forced: run hundreds or thousands of trials and define the best one as interesting. While I believe we are…
To show you the most relevant results, we’ve omitted some entries very similar to those already shown. Repeat the search with the omitted results included.