TL;DR
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
Z.ai’s GLM-5.3-FlashX-Legal scored 19.17% on Harvey’s Legal Agent Benchmark, placing seventh among 81 models on Vals AI’s leaderboard. The score counts a task as passed only when every required criterion is met; the same run passed 92.5% of individual criteria. Z.ai’s claim that it leads open-source models is harder to assess because the company did not specify the comparison set, and no published weights for the legal model were identified in the source material.
Z.ai’s GLM-5.3-FlashX-Legal scored 19.17% on the all-criteria pass measure of Harvey’s Legal Agent Benchmark, placing seventh among 81 models on Vals AI’s leaderboard. The result indicates that the model completed every required part of roughly one in five benchmark tasks; it does not mean it met only 19.17% of the individual criteria.
The leaderboard records the score as 19.17%, while Z.ai’s developer documentation rounds it to 19.2%. Vals AI lists the evaluation cost at $0.15 per test. On the same run, the model met 92.5% of individual criteria, according to the source material. That higher figure reflects partial success within tasks that still failed the benchmark’s all-or-nothing completion rule.
Z.ai says the legal model improved by 12.5 percentage points over its base model and achieved state-of-the-art performance among open-source models in this evaluation. The percentage-point improvement is consistent with the comparison on the same leaderboard, according to the source material. The open-source ranking claim is less clear: Z.ai does not name the comparison set, and the source author says they found no published weights for the legal model.
Z.ai describes GLM-5.3-FlashX-Legal as optimized for professional legal workflows, including document understanding, legal reasoning, research, contract review and drafting. Its documentation says the model was trained further on legal material and task environments modeled on legal work. Z.ai recommends using it as an agent connected to services for laws, cases and company data. The model is built on GLM-5.3-Flash, and Z.ai’s pricing page lists input tokens at $0.37 per million and output tokens at $1.25 per million.
A Closer Look at Z.ai’s Legal AI Benchmark Performance and Ranking
GLM-5.3-FlashX-Legal ranks seventh on Vals AI’s Harvey Legal Agent Benchmark. Its headline score measures fully completed tasks, while a much higher criteria score captures partial success inside tasks that still fall short.
Listed per test by Vals AI
Public tasks across 24 practice areas
Expert-written criteria
Z.ai-reported gain over base model
A high rank, with a demanding pass rule
Seventh place signals a strong relative position on this leaderboard. The 19.17% score answers a stricter question: how often did the model satisfy every requirement in a task? It completed roughly one in five tasks under that all-or-nothing rule.
Two measures, two meanings
A task can contain many successful checks and still fail overall when one required item is missed.
“A task is marked complete only if every criterion passes.”Harvey · Benchmark scoring approach
From matter file to reviewed work product
Harvey’s benchmark pairs an instruction with a client matter containing relevant and irrelevant materials. Agents produce a work product that is checked against binary rubric criteria such as facts, conclusions, citations, deadlines, recommendations and formatting.
| Evaluation detail | Public benchmark | Vals AI leaderboard run |
|---|---|---|
| Task set | 1,250 tasks, 24 practice areas | 120-task private holdout |
| Scoring inputs | 75,000+ expert-written checks | Two language-model judges |
| Internet access | Benchmark task environment | Disabled during evaluation |
| Reported result | Task-level rubric pass | Average of judges’ task pass rates |
Leaderboard performance is only one input
One holdout set
The 120-task evaluation does not establish performance across every jurisdiction, practice area, firm process or legal task type. No practice-area breakdown was identified in the source material.
Open-source comparison unclear
Z.ai claims state-of-the-art performance among open-source models in this evaluation, but does not name the comparison set. The source author found no published weights for the legal model.
Limited citation observations
A separate four-task check reported rulings the author could not find. These observations are limited to those tasks and do not establish broader error rates.
“A strong benchmark position is a reason to test against your own work—not proof of readiness for unsupervised legal use.”Practical interpretation
Built for legal workflows, with open questions
Z.ai describes GLM-5.3-FlashX-Legal as optimized for document understanding, legal reasoning, research, contract review and drafting. Its documentation says the model received further training on legal material and task environments modeled on legal work, and recommends connecting it to services for laws, cases and company data.
GLM-5.3-Flash
The legal model is built on Z.ai’s GLM-5.3-Flash base model.
$0.37 / $1.25
Per million input and output tokens, respectively, on Z.ai’s pricing page.
Application beta
AutoClaw announced an application-based beta on October 13, 2026. Vals lists the model release as October 12.
Questions firms should verify next
Before connecting a model to legal and company data, teams should check how outputs are sourced, reviewed and handled in their intended environment.
What does 19.17% measure?
The share of benchmark tasks that passed every required rubric criterion. A single failed check means the task does not count as complete.
Why is 92.5% different?
It is the share of individual criteria passed. Many checks can pass inside a task that still misses at least one requirement.
Does this show readiness without review?
No. The result is one evaluation on a limited holdout set. Firms should test representative work under their own review procedures.
What should deployment teams examine?
Verify retrieval sources, citations and dates; determine what client data reaches connected services; define professional review steps and data safeguards.
A High Rank With a Low Full-Pass Rate
The ranking and the pass rate answer different questions. Seventh place puts the model near the top of this leaderboard, but the 19.17% all-pass result shows how often it met every requirement in a task. In legal work, a missed citation, deadline or material fact may matter even when most of a document is accurate. Harvey’s scoring approach is designed to treat such omissions as failures of the complete work product.
For law firms and in-house legal teams, the result is a reason to examine how the benchmark maps to their own work, rather than a guarantee of readiness for unsupervised use. Z.ai frames outputs as drafts for legal professionals. Procurement and deployment decisions also depend on questions the score does not answer, including how client data is processed, what rules apply to that data, and who checks authorities and citations before a draft is used.
Products Worth Considering
LG UltraGear 27GX704A-B 27-inch QHD OLED Gaming Monitor
Logitech MX Master 4 for Mac Wireless Ergonomic Mouse, Space Black
Apple AirPods Pro 3 Wireless Earbuds with Active Noise Cancellation
As an affiliate, we earn on qualifying purchases.
How Harvey Scores Legal Tasks
Harvey released the Legal Agent Benchmark as open source on May 6, 2026. Its public set contains 1,250 tasks across 24 practice areas, assessed against more than 75,000 expert-written rubric criteria. Tasks pair an instruction with a client matter containing relevant and irrelevant materials, then ask an AI agent to produce a work product for review.
Harvey describes its criteria as binary checks covering items such as facts, conclusions, citations, recommendations, deadlines, dollar amounts and formatting. Under its all-pass grading, a task counts as complete only if every criterion passes. Harvey’s example of a change-of-control assignment has 57 criteria, illustrating why a small number of missed requirements can turn an otherwise substantial response into a failed task.
Vals AI’s public leaderboard uses a 120-task private holdout set that it says mirrors the public benchmark’s mix. Internet access is disabled during evaluation. Two language-model judges, GPT 5.5 and Claude Sonnet 4.6, grade each submission; the reported score averages their task pass rates. Z.ai’s AutoClaw blog announced an application-based beta on October 13, 2026, while Vals lists the model’s release as October 12. Z.ai’s documentation page does not carry a date.
“A task is marked complete only if every criterion passes.”
— Harvey, describing its benchmark scoring
Limits Beyond the Leaderboard
The score does not establish how the model performs across every legal task, jurisdiction or firm workflow. The benchmark’s 120-task holdout is one evaluation set, and the available source material does not provide a breakdown of GLM Legal’s results by practice area or task type. Z.ai’s documentation also does not specify the comparison group behind its open-source ranking claim.
The source author reports a separate four-task spot check, explicitly not a benchmark, in which the legal model cited rulings the author could not find. The author also says two Claude models cited a real ruling for the opposite of what it held, while the OpenAI models in that check invented none. These observations are limited to those tasks and do not establish broader model error rates. The source material does not establish how the models or citations were checked beyond the author’s account.
It is also unclear from the cited material where legal-model weights are available, how client data is handled in a particular deployment, or what safeguards a firm would need to configure when connecting the model to external legal and company data. Those details matter for evaluating practical and confidentiality risks alongside benchmark performance.
What Firms Should Verify Next
The immediate next step for prospective users is to review the leaderboard methodology and test the model against representative work under their own review procedures. Firms considering deployment will need to establish how legal sources are retrieved, how citations and dates are checked, what client information is sent to connected services, and which professional reviews are required before a draft is used.
Z.ai’s AutoClaw announcement describes a beta available by application. The source material does not state when broader access will begin or identify a further benchmark release date. Further clarity on model availability, the open-source comparison set and deployment terms would help firms judge how the leaderboard result applies to their intended use.
Key Questions
What does the 19.17% score measure?
It is the share of benchmark tasks that passed every required rubric criterion. A task with one or more failed criteria does not count as complete, even if the model passed most of its individual checks.
How did GLM-5.3-FlashX-Legal rank?
Vals AI’s leaderboard lists it seventh of 81 models, with a 19.17% all-criteria pass rate and a listed cost of $0.15 per test.
Why is the 92.5% criteria result different?
The 92.5% figure concerns individual criteria passed in the same run. The 19.17% figure counts only tasks in which all criteria passed, so a task can contribute successful criteria without counting as a completed task.
Does the benchmark show the model is ready to use without legal review?
No. The score reports performance on a defined benchmark and does not establish that the model can perform legal work without professional review. Z.ai describes its outputs as drafts for legal professionals.
What remains unclear about Z.ai’s open-source claim?
Z.ai says the model achieved state-of-the-art performance among open-source models in this evaluation, but the source material does not specify the comparison set. The source author also reports finding no published weights for the legal model.
Source: Thorsten Meyer AI
