… Although frontier models achieve relatively strong performance on direct expression evidence, performance drops substantially on functional, indirect, and weak-support evidence, with evidence-type confusion emerging as a dominant failure mode. …
… Notably, 17 out of 20 tested models perform worse than a random-guess baseline 15.2% on Gem Set, while the estimated human performance is ~81.1%, showing substantial room for improvement. …
… 2026 GUI-CEval: A Hierarchical and Comprehensive Chinese Benchmark for Mobile GUI Agents 2026 AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation 2026 Please give a thumbs up to this comment if you found it helpful! …
Papers arxiv:2605.12034 Boosting Omni-Modal Language Models: Staged Post-Training with Visually Debiased Evaluation Published on May 13 Submitted by liu on May 15 StepFun Authors: , , , , , , Abstract Research demonstrates that current omni-modal benchmarks may inflate performance through visual sh… …
… The following papers were recommended by the Semantic Scholar API MER-DG: Modality-Entropy Regularization for Multimodal Domain Generalization 2026 MissBench: Benchmarking Multimodal Affective Analysis under Imbalanced Missing Modalities 2026 Enhance-then-Balance Modality Collaboration for Robust M… …
… Similar to dataset contamination, where benchmark exposure leads to higher performance through memorization , we hypothesize that models trained on texts describing evaluation practices may implicitly learn to recognize and respond to evaluation-like contexts, for instance, through exposure to scie… …
… The following papers were recommended by the Semantic Scholar API KernelSkill: A Multi-Agent Framework for GPU Kernel Optimization 2026 AutoKernel: Autonomous GPU Kernel Optimization via Iterative Agent-Driven Search 2026 KernelFoundry: Hardware-aware evolutionary GPU kernel optimization 2026 AdaEx… …
… Evaluation of six state-of-the-art methods shows that strong global volumetric overlap does not necessarily translate into reliable performance at clinically critical tumor-vessel interfaces. …
Papers arxiv:2605.03596 Workspace-Bench 1.0: Benchmarking AI Agents on Workspace Tasks with Large-Scale File Dependencies Published on May 5 Submitted by taesiri on May 6 Authors: , , , , , , , , , , , , , , , , , , , Abstract Workspace-Bench is a benchmark for evaluating AI agents on workspace lea… …
… Automated Auditing of LLM Agent Benchmarks 2026 Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents 2026 ClawArena: Benchmarking AI Agents in Evolving Information Environments 2026 AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation 2026 OccuBench: Evaluating AI Agen… …