While model and agent evaluation are inextricably linked, their technical benchmarks and metrics for success are fundamentally different.
Mastering Agentic Techniques: AI Agent Evaluation | NVIDIA Technical BlogPapers arxiv:2606.00285 Model-Based Quality Assessment for Massively Multilingual Parallel Data Published on May 29 Submitted by Shaoxiong on Jun 2 MaLA-LM Authors: , Zihao Li , , Abstract Multilingual parallel-data…
…for Evaluating Agent Values Published on May 11 Submitted by Haoran Ye on May 13 Peking University Authors: Haonan Dong , , , , , Abstract Autonomous agents exhibit distinct value systems from underlying language models, requiring…
…data by sampling the model on each of the prompts and filtering down to cases where the assistant chose not to take the honeypot. Despite very closely matching the evaluation distribution, we…
…We evaluate 13 frontier models under a unified protocol. Key findings: The top model (Claude Opus 4.6) passes only 66.7% of tasks, and no model reaches 70%. Local workspace repair…
Two years ago we released VQAScore: ask a VLM "does this image show {prompt}?" and use P(Yes) as the score. It became a go-to evaluation metric and reward model for image generation, replacing CLIPScore across the field …
Current LLM benchmarks are broken. We think long horizon "world" building could be an interesting additional way to evaluate LLMs, since it combines many aspects such as need for advanced reasoning, tool calling, working…
I built AptSelect to stop writing throwaway scripts every time I needed to test how different LLMs handle specific instructions and prompt edge cases.What it does:Parallel Execution: Send a single prompt to OpenAI, Anthr…
I'm excited to announce Elmo, an MIT-licensed, open source AEO/GEO tool.We help you scrape ChatGPT/Google AI Mode/etc using web scrapers like BrightData/Olostep/etc, evaluate prompts against the OpenAI/Anthropic/Mistral …
TLDR: Forked pytorch and triton internals . Changed attention so its linear first layer , middle quadratic layer, last linear layer Inference got much faster with a low perplexity hit in tests .Full attention O(n²): 17.9…
…Do VLMs Know When Not to Answer Spatial Questions (and Why)? Published on May 28 Submitted by Yue Zhang on Jun 1 Authors: , , , , , Abstract Vision-language models exhibit overconfidence in spatial reasoning…
…Rather than treating urban imagery as generic scene data, Urban-ImageNet evaluates whether machine perception models can capture spatial, social, and functional distinctions that are central to urban studies. The benchmark supports…
…foundation generative world model trained from the Cosmos diffusion model, enables real-time action-conditioned video generation for autonomous driving policy evaluation in complex, unseen scenarios. Generated by Qwen/Qwen2.5-Coder…
…is introduced to systematically evaluate speech modification capabilities across atomic and compositional tasks. Generated by Qwen/Qwen2.5-Coder-32B-Instruct Instruction-guided speech editing requires a model to modify specified speech…
…of model capabilities across heterogeneous evaluation settings, a signal rich enough to learn from directly. By learning a performance-aware latent space over model--dataset--metric tuples , ModelLens ranks unseen models on…
…Social deduction games have become a popular testbed for evaluating reasoning, deception, coordination, and belief modeling in large language models. However, most environments evaluate agents primarily through game outcomes like win rates…