法律 AI // 资讯总览
需要被持续追踪的
法律 AI 信号。
本期索引按监管、判例、实务与研究归档;可依主题或关键字定位已发布记录。
法律 AI // 资讯总览
本期索引按监管、判例、实务与研究归档;可依主题或关键字定位已发布记录。
原题: Cherry-pick Override: LLM Judges Under-use the Non-Directional Verdicts Their Contract Authorizes
研究以AVeriTeC金标Conflicting子集与VitaminC-Mixed压力测试检验四款冻结LLM法官。即便提示词详述冲突、禁止塌缩并给出示范,仍有18.7%至48.0%呈方向性过度承诺;CPO指两方证据俱在却只采其一,补充检索无法修复。证据不足案件的越权与CPO相当或更高;换语境后,两个法官能识别约三分之二自身CPO。类型化词表、投票、阈值、结构校验与自分解均留残差。
展示来源全文,完整内容以原文页面为准。
arXiv:2606.07834v2 Announce Type: replace-cross Abstract: Evidence-grounded fact verification increasingly uses LLM judges to turn evidence into terminal verdicts. Many task contracts deliberately include non-directional verdicts - Conflicting for materially mixed evidence, Not Enough Evidence for absent evidence - so that a system can decline to assert a direction. Returning Supports or Refutes on a claim whose authorized verdict is non-directional is directional overcommitment: a label error whose output is a direction the contract did not authorize. Its conflict instance - both strands present, one of them returned - is Cherry-pick Override (CPO); unlike the insufficiency instance it cannot be repaired by retrieving more, because nothing was missing. On AVeriTeC's gold-Conflicting subset (N_C=150) a four-option typed panel commits directionally on 18.7% of claims, and four frozen contemporary judges (two proprietary, two open-weight) on 21.3-48.0%, with 24.0-38.0% on VitaminC-Mixed, a constructed stress test on which the ranking reverses. These are conservative lower bounds: the shared prompt specifies Conflicting in detail, forbids collapsing to a direction, and demonstrates the label three times. The failure is not specific to conflict: overcommitment on gold-Insufficient cases runs at a comparable rate for every judge and exceeds CPO for three of four on VitaminC-Mixed. Asked in a fresh context, two judges identify the conflict on about two thirds of their own CPO claims, against under 0.12 where they committed correctly: the recognition is available and does not reach the terminal action. Typed vocabulary, panel voting, confidence thresholds, a structural validator and self-decomposition each leave a substantial residual, and the last buys part of its reduction by relabelling one-sided evidence as conflicting. Every rate is dataset-defined; we report no human-confirmed prevalence.
自动化事实裁决在证据冲突或不足时倾向方向性定案,可能架空非方向性结论与双方证据的平等评价。
文书与研报规范引注
《大模型法官少用授权的非方向裁决》(来源: arXiv cs.CL,发布日期: 2026/09/15,访问链接: https://arxiv.org/abs/2606.07834)