多语言刑事法律中LLM过度对齐的测量与缓解
SOURCE / arXiv cs.CL · Measuring & Mitigating Over-Alignment for LLMs in Multilingual Criminal Law Courts
原文
Measuring & Mitigating Over-Alignment for LLMs in Multilingual Criminal Law Courts
完整原文
arXiv:2606.23375v2 Announce Type: replace Abstract: While the wider applicability of LLMs in the legal field is currently debated due to their reliability and the gravity of any errors, narrow uses with well-understood and mitigated risks have emerged. Notably the Swiss Federal Supreme Court uses small on-premises models for tentative translations and short-passage summarization across the four official languages. However, such usage is challenging in the context of Criminal Law. Since rulings and cases employees work on routinely can contain detailed descriptions of violent and sexual offenses, their legitimate work is compromised by refusals and disclaimers due to the activation of model guardrails (over-alignment). To measure this phenomenon, we introduce TF-RefusalBench, a multilingual benchmark for criminal-law translation and summarization derived from public Swiss Supreme Court rulings. TF-RefusalBench contains 5,200 total prompts across French, German, Italian, and English, corresponding to common task prompts and passages likely to trigger refusal. We then use TF-RefusalBench to show that over-alignment is a multifaceted phenomenon, influenced by the model and the prompt and text languages being processed, and that its impact cannot be evaluated solely from an over-refusal perspective, given the disclaimer's impact on task faithfulness. Finally, we evaluate approaches to enable on-premises LLMs for Criminal Law Tasks, demonstrating that while prompting can be effective, abliteration (refusal directions ablation) eliminates refusal with minimal impact on task performance.
归纳
针对大型语言模型在刑事法律领域因模型护栏导致拒绝和免责声明（过度对齐）的问题，研究引入TF-RefusalBench多语言基准测试，该基准源自瑞士联邦最高法院公开裁决，包含法语、德语、意大利语和英语共5200个提示，涵盖易触发拒绝的常见任务。实验表明，过度对齐受模型、提示及文本语言影响，且其影响不能仅从过度拒绝角度评估，还需考虑免责声明对任务忠实度的损害。研究评估了提示工程和abliteration（消除拒绝方向）两种缓解方法，后者能消除拒绝行为且对任务性能影响最小。
点评
LLM在刑事法律场景中的过度对齐可能导致法律分析失真，构成对用户法律咨询义务的潜在违反。
法律视角点评
AI 生成 · 人工审核核心关切
LLM在刑事法律场景中的过度对齐可能导致法律分析失真，构成对用户法律咨询义务的潜在违反。
实务启示
中国法律人应警惕AI法律工具因过度安全限制而输出不完整或误导性意见，需在合规框架内验证模型输出的忠实度。