TL;DR
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
A test of 42 AI model setups on 15 legal questions produced scores from 0 to 79, with Claude Opus 5.5 leading and a law-focused GLM model scoring 8. The test operator says graders were Claude agents and discloses that Claude ranked first; the results are a limited spot check, not a broad benchmark.
Thorsten Meyer AI has published results from 42 AI model setups answering 15 legal questions, reporting scores from 0 to 79 out of 100. Claude Opus 5.5 scored highest at 79, while GLM-5.3 FlashX Legal, marketed by its maker as optimized for legal workflows, scored 8; the publisher says the test is a limited spot check and discloses that Claude agents graded the answers.
The Lexassay test covered German, US and EU law, including deadline calculations, recent legal changes, court decisions and questions built around false premises. Models answered without internet access or tools. The test included ten public questions and five hidden ones; only scores for the hidden tasks are published, according to the description of the method.
Among the reported results, GPT-6 Astra scored 72, GPT-6.1 Sol 71 and Claude Sonnet 5.5 69. Mistral Large 4 scored 10, while GLM-5.3 FlashX Legal scored 8 and fully solved none of the 15 tasks, according to the publisher. Two of its answers were left blank. These figures come from the published test and do not establish how the systems would perform with document retrieval, legal tools or different questions.
The publisher says graders compared responses with reference answers containing weighted criteria and checked citations against official databases. Across published runs, Lexassay says it verified 2,644 citations as real and identified 804 as invented or as failing to support the model’s claim. The grading rules deduct points for inaccurate citations and for answers that omit a legal-advice disclaimer or recommendation to consult a lawyer.
A 42-Setup AI Test Reveals A Surprising Legal Score
Lexassay tested 42 AI setups on 15 legal questions. The scores show how results varied under this test’s no-tools conditions, not how every system performs across legal work.
Scores from the spot check
The published results span a wide range on these questions.
What the test covered
Models answered without internet access or tools.
Three legal systems
Questions covered German, US and EU law.
Practical legal questions
Tasks included deadline calculations, legal changes, court decisions and false premises.
Citations and cautions
Grading deducted points for inaccurate citations and missing legal-advice disclaimers or lawyer recommendations.
Why a deadline answer matters
The test’s example shows how a public holiday can affect a German civil appeal deadline.
- Appeal served April 27, 2027
- Appeal deadline: May 28, because May 27 is a public holiday in Bavaria
- Appeal served April 27, 2027
- Appeal deadline: May 27
Limits to keep in mind
The scores describe a narrow test, with one answer per model setup for each task.
- Different effort settings were tested separately, and setups may appear more than once.
- Small score differences may reflect noise; the publisher says this is not a strict benchmark.
- No internet, tools or supplied legal documents were tested.
- Claude agents graded the answers, and Claude ranked first; the publisher discloses this potential conflict.
Citation Errors Can Change Outcomes
The test’s deadline example shows why errors can have practical consequences. In a scenario involving German civil appeals served on April 27, 2027, the publisher says the appeal deadline would fall on May 28 in Munich because May 27 is a public holiday in Bavaria, but remain May 27 in Berlin. The stated deadline for written grounds would be June 28 in both cities. The example is part of the test material, not a report of an actual court case.
Missing a filing deadline can put a case at risk, while a citation that exists but does not support the stated proposition can make a legal answer appear more reliable than it is. The results therefore speak to a practical limitation of standalone chatbots: fluent answers and legal product claims do not, by themselves, establish that a response is accurate enough for a filing or deadline.
Lexassay advises readers to treat AI output as a starting point, check cited laws and rulings against their sources, and consult a lawyer when a deadline, financial stakes or an authority is involved. The test does not assess a lawyer’s review process or every product feature offered by the companies.
Products Worth Considering
LG UltraGear 34GX90SA-W 34-inch WQHD Curved OLED Gaming Monitor
Logitech MX Mechanical Wireless Illuminated Keyboard, Tactile Quiet, Graphite
Logitech MX Master 4 Ergonomic Wireless Mouse with Haptic Feedback, Graphite
As an affiliate, we earn on qualifying purchases.
A No-Tools Test With Limits
Lexassay describes its assessment as a spot check, with one answer per model setup for each task. It tests different effort settings separately, so a model can appear more than once. The publisher cautions that small score differences may reflect noise and says the exercise is not a benchmark in the strict sense.
The results also differ from another assessment cited by the publisher. Vals AI’s legal-agent benchmark supplies models with documents; its ranking places Mistral Large 4 eighth, ahead of Claude Opus 5.5 in 21st place and GPT-6 Astra in 24th. Because that test uses supplied documents and a different method, its rankings are not directly comparable with Lexassay’s no-internet, no-tools questions.
Thorsten Meyer AI says it runs Lexassay and that Claude agents graded responses blind. It also notes that Claude ranked first, a potential conflict of interest. The site says it publishes answers, reference answers and citation checks so readers can inspect the grading. The article is dated November 4, 2026, and says it is not legal advice.
How Far the Scores Apply
The results do not show how consistently each setup would answer the same questions across repeated runs: the publisher says there was one answer per model and task. The test also excludes internet search and tools, so it cannot establish how systems perform when they can retrieve current law or work from supplied documents.
The source does not provide enough information here to independently assess every grading decision, the full distribution of errors by topic, or whether the results generalize to other legal questions and jurisdictions. The disclosed use of Claude agents as graders and Claude’s top ranking are relevant when interpreting the scores. The citation totals are reported by Lexassay, which runs the test.
Check the Published Answers
Readers can review Lexassay’s published method, answers, reference answers and citation checks, according to Thorsten Meyer AI. Those materials may help clarify how points were assigned and which citations were judged inaccurate. The source article does not announce a date for a new test or a change to the current question set.
For anyone using chatbot output in a legal matter, the immediate step is to verify cited law and court decisions at the source and have a lawyer assess deadlines or consequential decisions. The test’s findings remain limited to its questions, scoring method and no-tools conditions.
Key Questions
What did the Lexassay test measure?
It scored 42 AI setups on 15 questions about German, US and EU law. Models answered without internet access or tools, according to the publisher.
Which setup scored highest?
Claude Opus 5.5 scored 79 out of 100, followed by GPT-6 Astra at 72 and GPT-6.1 Sol at 71. The publisher discloses that Claude agents graded the answers and that Claude ranked first.
How did the law-focused GLM model perform?
GLM-5.3 FlashX Legal scored 8 out of 100 and fully solved none of the 15 tasks, according to Lexassay. The result applies to this test’s questions and conditions.
Can these scores predict performance on other legal work?
Not on their own. Lexassay describes the results as a spot check with one answer per model and task. It did not test internet access, tools or document-based legal work.
Should I rely on a chatbot for a legal deadline?
The publisher advises readers to verify every citation and not take a deadline from a chatbot. For a deadline, contract or letter to an authority, it recommends consulting a lawyer.
Source: Thorsten Meyer AI
