法律 AI // 资讯总览
需要被持续追踪的
法律 AI 信号。
本期索引按监管、判例、实务与研究归档；可依主题或关键字定位已发布记录。
法律 AI // 资讯总览
本期索引按监管、判例、实务与研究归档；可依主题或关键字定位已发布记录。
原题: Do Large Language Models Know Colombian Law? A Reliability Benchmark for the Colombian Legal System
涉及法律行业大模型应用、律所与企业法务数智化实践指引。
发表于arXiv的一项预印本研究评估了大语言模型处理哥伦比亚法律问题的可靠性。研究团队构建了由专家验证的1042道题目数据集，覆盖10个法律领域，并设置封闭式选择题、半开放题和开放式IRAC问答三种任务形式，对15个商业模型及开放权重模型进行测试。封闭式问题准确率介于0.577至0.905之间，但自由文本法律回答的事实正确性最高不超过0.45。研究还发现，回答的相关性与正确性并不一致，二者的Spearman相关系数为-0.46，说明模型可能生成听起来切题但包含事实错误的答案。模型引用的法律规范约半数正确，其余可能错误或不存在。选择题成绩与自由文本表现具有较高相关性，但会高估模型的实际法律可靠性。研究建议在哥伦比亚法律场景中继续实行专家监督，并结合权威法律来源开展检索增强。该成果是未经arXiv同行评审的评测性学术研究，不构成哥伦比亚法律规范、司法裁判或执法要求。
展示来源全文，完整内容以原文页面为准。
View PDF
HTML (experimental)
Abstract:Large language models (LLMs) are increasingly used to support legal practice, education, and research, yet their reliability in national legal systems outside the United States remains largely undocumented. We introduce an expert-validated benchmark for evaluating LLM reliability on the Colombian legal system. The benchmark comprises 1,042 items spanning ten areas of law and three question formats (closed multiple-choice, semi-open, and open-ended IRAC), built through a human-in-the-loop pipeline with multi-stage expert review. We evaluate 15 contemporary proprietary and open-weight models with format-appropriate metrics. Accuracy on closed questions ranges widely, from 0.905 (Gemini 3.1 Pro) to 0.577, but on free-text legal answers factual correctness never exceeds 0.45 (on a 0-1 scale) for any model. We find a dissociation between answer relevancy and correctness (Spearman rho = -0.46): models reliably sound responsive while frequently being wrong, a pattern of particular concern for non-expert users. Closed-question accuracy and free-text correctness are strongly rank-correlated (rho = 0.94), so cheap multiple-choice screening predicts model ranking but overstates absolute reliability. An independent rubric-based LLM judge and blind human expert scoring both reproduce the free-text ranking (rho >= 0.88). The judge further reveals that only about half of the norms models cite are correct; the rest are wrong or non-existent. Reliability varies systematically by legal area and follows an inverted-U across question complexity. Our results indicate that current LLMs require expert supervision for Colombian legal tasks, and that grounding answers in authoritative sources is a promising path to higher reliability. We release the benchmark construction pipeline to support reproducible evaluation.
Submission history From: Rubén Manrique [view email] [v1]
Fri, 2 Oct 2026 17:27:41 UTC (292 KB)
文书与研报规范引注
《哥伦比亚法律大模型可靠性基准研究》（来源: arXiv cs.CY，发布日期: 2026/10/05，访问链接: https://arxiv.org/abs/2610.03639）