… 0.793 semantic agreement vs. 0.791 surface-form self-consistency Across Llama-3.1-8B, Mistral-7B-v0.3, and Qwen2.5-7B on PopQA and TriviaQA n=1000 each Ensembling ϕ first with semantic agreement adds only +0.02 AUROC — first-token confidence already carries most of the signal Feedback welcome. …
… We introduce a unified taxonomy of annotation-reporting practices and validate an LLM-assisted extraction pipeline against Annotated-gold, a human-adjudicated gold standard of 41 papers and 72 annotation tasks, where the best model reaches human-comparable agreement with adjudicated labels, with Kr… …
… For reasoning-style analysis, we instantiate it as an Answer Agreement Verifier that groups trajectories by answer agreement and uses self-consistency as an auxiliary signal. …
… System-expert agreement was quantified using ICC 2 , 1 and linearly weighted Cohen's kappa , benchmarked against the human inter-rater baseline. …
… Because no labels are available, we replace ground-truth agreement with an instrumental-validity chain: responsiveness to a controlled safe-versus-abliterated contrast, dominance of target-driven variance over auditor and judge artifacts, and stability across reruns. …
… Existing sycophancy research primarily focuses on factual agreement and instruction-following settings, leaving culturally grounded conversational sycophancy underexplored. …
… We first ask whether such scores faithfully capture comparative preference: in a controlled study with eight expert annotators, score-derived rankings align poorly with the same annotators' direct comparisons, while direct ranking yields substantially higher inter-annotator agreement on best- and w… …
… UniSD integrates complementary mechanisms that address supervision reliability, representation alignment, and training stability, including multi-teacher agreement , EMA teacher stabilization , token-level contrastive learning , feature matching , and divergence clipping . …
… We also construct the DeltaScene Dataset , a large-scale dataset generated through an automated pipeline with 3D agreement filtering to ensure ground-truth quality. …
… Experiments show that TrOPD consistently outperforms SoTA OPD baselines, including OPD, EOPD, and REOPOLD, across mathematical reasoning, code generation, and general-domain benchmarks. the bit i keep coming back to is the top-k forward kl estimator for outliers in troupd. by quantifying teacher-st… …