🔍 Read the full analysis: FrontierCode: Your AI Opened A Pull Request—Would A Maintainer Merge It? on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Artificial Analysis added FrontierCode to its evaluations on October 12, 2026, and tested 23 models on tasks designed to measure whether maintainers would accept an AI-written change. Claude Opus 5.5 led with a 52.2% overall score and passed every blocking check in 57.7% of attempts; the results do not establish that any model can reliably submit merge-ready pull requests without human review.
Artificial Analysis added FrontierCode results on October 12, 2026, after testing 23 AI models on coding tasks judged by whether changes satisfy a range of maintainer-style checks. Claude Opus 5.5 ranked first with a 52.2% score, but it passed all blocking checks in 57.7% of attempts, showing that even the leading model did not consistently produce patches meeting the benchmark’s acceptance criteria.
FrontierCode evaluates more than whether a patch makes tests pass. Its checks cover hidden tests, command-line checks, test quality, scope limits and an LLM-based review against criteria set by maintainers. A failed blocking check results in a zero for that attempt; passing them all earns a score from 70% to 100%, depending on additional quality checks. The benchmark can also assign zero if its trajectory scanner flags an attempt for looking up a solution online.
In Artificial Analysis’ results, GPT-6.1 Sol scored 49.1% and Claude Fable 5.1 scored 49.0%, just behind Opus. The reported pass rates for those models were 54.7% and 54.3%, respectively. Xiaomi’s MiMo-V2.6-Pro scored 45.4%, while DeepSeek V4.1 Flash scored 44.0% and Kimi K3 scored 43.6%. Nine of the benchmark’s 100 tasks in the evaluated Main set were not solved by any model on any attempt.
Artificial Analysis ran models using the same open-source agent harness, Stirrup, with a limit of 250 turns per attempt and up to three attempts per task. The LLM review step used GPT-6 Luna (medium) as judge. Results therefore describe performance under this benchmark’s particular setup, not a guarantee of how a model will perform in every repository or development workflow.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Why Maintainer-Style Checks Matter
The results address a practical question for teams using coding agents: does a change meet standards beyond passing tests? FrontierCode includes checks for regressions, whether tests meaningfully catch faulty behavior, and whether a patch stays within an appropriate scope. Those criteria target common sources of review burden that a simple green test suite may not reveal.
The figures also put limits on claims about autonomous pull requests. The leading model passed all blocking checks in 57.7% of attempts, not all of them, and nine tasks resisted every model attempt in the evaluated set. That does not show that AI tools cannot help with code; it does mean these results do not support treating benchmark performance as evidence that human review can be removed.
Price varied sharply as well. Artificial Analysis reported $0.09 per task for GPT-6 Luna (max), which scored 41.9%, and $10.15 for Opus, which scored 52.2%. GPT-6.1 Sol scored 49.1% at $0.81 per task. These figures offer a benchmark-specific comparison of cost and score; they do not include every possible deployment expense or establish which model is cheapest for a particular team’s work.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Tests Patches
Cognition developed FrontierCode with open-source maintainers. The benchmark was announced in June 2026, followed by version 1.1 in July, which introduced rules against finding solutions online. According to the source material, more than 20 maintainers created tasks drawn from 36 repositories, including Celery, Budibase, uppy and Mattermost, spending more than 40 hours on each task.
The full benchmark contains 150 tasks; Artificial Analysis ran its 100-task Main set. Tasks are private to limit the chance that models have encountered their solutions during training. The grading combines automated checks with an LLM code review. As Celery maintainer Tomer Nosrati put it in Cognition’s announcement, “Where others grade like a CI, FrontierCode grades like a tech lead.” That description reflects the benchmark’s focus on whether a patch is acceptable as a change, rather than merely whether it passes tests.
Artificial Analysis’ comparison also found that general model rankings do not map neatly onto FrontierCode performance. For example, Claude Sonnet 5.5 ranked fifth on FrontierCode despite ranking second among the models shown in the Intelligence Index comparison; Grok 4.7 scored 33.8% and ranked 18th on FrontierCode. The different rankings suggest that broad capability scores alone do not settle how well a model handles this benchmark’s coding and review criteria.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, a Celery maintainer, in Cognition’s announcement
software development code review software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Limits of the Published Results
The reported scores are specific to the 100-task Main set, the Stirrup harness, its turn limits and the benchmark’s review setup. The source material does not provide enough detail to determine how scores would change with different agent tools, prompts, repositories or review standards. Benchmark results also do not establish how often a model’s patch would be merged in ordinary software projects.
Several models had no FrontierCode result as of October 13, including Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview. Their eventual performance could change the comparison. The available material also does not give results for every cost point or explain how the benchmark’s scores translate into productivity, review time or software reliability in deployed teams.
As an affiliate, we earn on qualifying purchases.
More Models and Comparisons Ahead
The immediate next step is to watch for FrontierCode results from models that were missing from the October 13 comparison, particularly those listed by the source as having no score. Artificial Analysis may update its evaluations as new results become available, but the source material does not give a publication schedule.
For maintainers and engineering teams, the results provide a reason to compare coding models on repository tasks and review standards relevant to their own work, rather than relying only on general capability rankings. Whether an AI-generated pull request is suitable to merge will still depend on the change, the project’s requirements and human review; FrontierCode’s current results do not settle that question for any specific codebase.
As an affiliate, we earn on qualifying purchases.
Key Questions
What does FrontierCode measure?
It scores AI coding agents on repository tasks using checks for correctness, regressions, test quality, patch scope and an LLM review against maintainer criteria. A failed blocking check gives the attempt a zero.
Which model ranked first?
Claude Opus 5.5 (max with fallback) led Artificial Analysis’ results with a 52.2% FrontierCode score and passed every blocking check in 57.7% of attempts.
Does the benchmark show that AI pull requests are ready to merge?
No. It reports performance on a defined set of tasks and checks. Even the leading model failed to pass all blocking checks in a substantial share of attempts, and the results do not replace review of a real change in its project context.
How many tasks and models were included?
Artificial Analysis tested 23 models on FrontierCode’s 100-task Main set. The full benchmark contains 150 tasks, according to the source material.
Which models were missing from the results?
As of October 13, 2026, the source listed Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview as having no FrontierCode result.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
