🔍 Read the full analysis: Z.ai’s Legal AI Benchmark Rank And Pass Rate: Why They Don’t Tell The Same Story on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Z.ai’s GLM-5.3-FlashX-Legal scored 19.17% on Vals AI’s Harvey Legal Agent Benchmark leaderboard, placing seventh among 81 models. The figure counts a task as passed only when every rubric criterion is met; Vals separately reports a 92.5% pass rate for individual criteria. The results show why a model’s rank and task-level completion rate can tell different stories, but neither establishes how safely it will perform in a law firm.
Z.ai’s GLM-5.3-FlashX-Legal ranked seventh of 81 models on Vals AI’s Harvey Legal Agent Benchmark leaderboard, with a 19.17% all-criteria task pass rate, as detailed in a closer look at its benchmark performance. That score means the model met every required criterion on roughly one in five test tasks; Vals also reported that it passed 92.5% of individual criteria, a different measure that helps explain why the two numbers tell different stories.
The benchmark’s task-level score is deliberately strict. Harvey says a task counts as complete only if the system passes every criterion in its rubric. A task can include requirements about facts, conclusions, citations, deadlines, recommendations and formatting. Missing even one required item means the entire task does not pass, regardless of how many other criteria the model satisfies.
Vals AI’s archived leaderboard lists GLM Legal at 19.17% of tasks, at a reported cost of $0.15 per test. The listing places it behind six models and ahead of dozens of others. Vals reports a separate 92.5% criteria pass rate; in that leaderboard version, a criterion counted only when both AI judges passed it. The 19.17% and 92.5% figures measure different things and should not be treated as competing estimates of the same outcome.
Z.ai’s documentation describes the model as designed for legal research, document understanding, contract review and drafting. The company says it underwent legal-domain mid-training and post-training, including reinforcement learning in task environments modeled on legal workflows. Z.ai’s announcement says the benchmark result improved by 12.5 percentage points over the base GLM 5.3 Flash model. The archived table supports that comparison: 19.17% for GLM Legal versus 6.67% for the base model. Z.ai also calls the result state of the art among open-source models, but the supplied material does not identify its comparison set, and published weights for the legal model were not found by the source author.
Why Task Completion Matters
The split between the scores matters to anyone weighing legal AI for professional work. A high rate of passing individual rubric criteria suggests the model can often get parts of a task right. But a low all-criteria rate indicates that, under this benchmark’s rules, it frequently leaves at least one required element incomplete. For legal work, a missed citation or deadline may matter even when the rest of a draft is useful.
The leaderboard is useful for comparing systems under a shared evaluation, not as a guarantee of courtroom reliability or firm-ready performance. Its rank also does not erase the underlying result: seventh place is relative, while 19.17% is an absolute task pass rate. Buyers still need to examine the nature of errors, how outputs are checked, and the handling of confidential client information before deciding whether to use a model.
Cost is another part of the comparison, though not a complete measure of value. Vals lists GLM Legal at $0.15 per test, substantially below the listed prices for several higher-ranked entries. That makes the score relevant to procurement discussions, but low benchmark cost does not establish the full cost of human review, integration, data controls or correcting errors.
Products Worth Considering
Bose S1 Pro+ All-in-One PA Speaker
Apple AirPods Pro 3 Wireless Earbuds with Active Noise Cancellation
DJI Osmo Mobile 7P Phone Gimbal with Mic Mini Transmitter, Infinity Black
As an affiliate, we earn on qualifying purchases.
How Harvey Scores Legal Tasks
Harvey released its Legal Agent Benchmark as open source on May 6, 2026. It contains 1,250 tasks across 24 legal practice areas, with more than 75,000 expert-written rubric criteria. Each task presents an instruction and client matter materials, then asks an agent to produce work for review. Harvey says the leaderboard evaluates submissions on a private holdout set designed to reflect the public task mix.
According to Vals AI’s description, its holdout contains 120 tasks, internet access is disabled, and two language-model judges score each submission. The final task pass rate averages the judges’ task-level results. The archived Version 1 leaderboard was updated October 18, 2026; Z.ai’s AutoClaw blog announcement is dated October 13, while Vals lists the model release as October 12. Z.ai’s documentation page itself carries no date.
Harvey’s earlier baseline provides a limited point of reference, not a direct comparison. In a May 26 assessment, Harvey reported that the best all-pass score among the frontier models it tested was 7.1% and said those models completed fewer than 10% of tasks end to end in aggregate. The tests were separate runs, so those figures should not be read as a like-for-like ranking against GLM Legal.
“A task is marked complete only if every criterion passes.”
— Harvey, describing its grading rule
What The Scores Leave Open
The leaderboard does not show how GLM Legal performs across the full range of real client matters, nor does its aggregate score reveal which kinds of errors account for failed tasks. The available source material does not provide detailed task-by-task results or establish how the benchmark’s errors compare with the risks a particular firm faces.
The source author reports a small, four-task spot check in which GLM Legal cited rulings the author could not find. The author also says that OpenAI and Claude models in the same informal check did not invent rulings, while two Claude models cited a real ruling for the opposite of its holding. This is an informal sample, not a benchmark, and it cannot establish comparative error rates. It does underscore that citations and legal conclusions require verification.
It is also unclear from the supplied material what client-data protections apply across Z.ai’s services, which jurisdictions govern processing, or what human review controls accompany deployment. The source author says Z.ai’s pages describe outputs as drafts for legal professionals, but reports finding no “not legal advice” notice on the pages reviewed. Those observations do not establish the terms or safeguards for every deployment.
Checks Firms Need Before Use
For law firms considering the model, the next steps are practical: review Z.ai’s current service terms and data-processing arrangements, test the model on representative work, and set a process for checking every citation, deadline and material conclusion before a draft is used. Z.ai recommends running the model as an agent connected to services for laws, cases and company information, so firms would also need to examine the sources and permissions involved in those connections.
Further updates to the Vals leaderboard or more detailed evaluations could clarify how GLM Legal compares across benchmark versions and task types. Until then, the reported evidence supports a bounded conclusion: GLM Legal ranked seventh on the cited leaderboard, passed about one in five tasks under all-criteria grading, and passed a much larger share of individual criteria. How that performance translates into safe and useful legal work remains a question for controlled testing and human review.
Key Questions
What does GLM Legal’s 19.17% score mean?
It is the share of benchmark tasks on which the model met every required criterion, according to Vals AI’s archived leaderboard. A task with even one failed criterion does not count as a pass.
Why is the criteria pass rate 92.5%?
That is a separate measure of individual rubric criteria rather than whole tasks. Vals says a criterion counted only when both judges passed it. One task may include many criteria, so a model can pass most criteria while failing at least one on many tasks.
Was GLM Legal seventh among all legal AI systems?
It was listed seventh among 81 models on the archived Vals AI leaderboard cited in the source material. That rank applies to that leaderboard version and evaluation, not to every legal AI system or real-world use case.
Does the benchmark show the model is safe for law firms?
No. The result is a benchmark score, not a guarantee of reliable legal work. Firms would still need to check outputs, citations, privacy terms and data handling before using the system with client matters.
Source: ThorstenMeyerAI.com
