TL;DR
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
A 26-page preprint reports that Terminal-RSI-27B raised its score on Terminal-Bench 4.0 from 0% to 19.84% through five training stages. Researchers designed and ran those stages, using verified successful attempts and reinforcement learning; the results do not show an AI improving itself without human involvement.
A 26-page preprint called Terminal-RSI reports that a 27-billion-parameter terminal agent improved from 0% to 19.84% on Terminal-Bench 4.0 after five training stages. The work, from Tencent’s Hy Frontier lab and four universities, describes a pipeline built and managed by researchers—not an AI that independently rewrites and retrains itself without human direction.
The pipeline alternates between learning from verified successful attempts and reinforcement learning. In the first method, called Recursive Self-Rewrite, the model’s accepted attempts are turned into step-by-step plans, screened for leaked answers, and run again from scratch. Only plans that pass the verifier again become training examples. The reinforcement-learning stages then adjust the model based on scored attempts, with a critic network estimating their quality.
The sequence was Base, RSR-1, RPR-1, RSR-2 and RPR-2, starting from the open Qwen3.8-27B model and ending at a checkpoint named Terminal-RSI-27B. People selected the stages, rewards and learning rates, ran the training and chose which checkpoints to keep. The work drew on a pool of 3,000 tasks.
The paper reports an equal-weight average across four Terminal-Bench suites of 50.58% for the final model, compared with 20.68% for the base model. Each reported score is the mean of three evaluation runs. On a long-horizon benchmark, the final model completed an average of 5 of 46 tasks, compared with zero for the base model. These results are benchmark measurements, not evidence that the system can improve itself without people.
Terminal-RSI What the results show
A researcher-run training pipeline improved benchmark scores. The preprint does not show an AI improving itself without human involvement.
People directed the process
Researchers selected the stages, training settings and checkpoints.
Started from the open Qwen3.8-27B model.
Used verified attempts to create plans, then checked them again.
Reinforcement learning adjusted the model using scored attempts.
Repeated the verified self-rewrite stage.
Completed the pipeline as Terminal-RSI-27B.
Verification changes what counts
The agent worked in a command line, where claiming completion did not prove the task was done.
- Failed attempts received 50.09% of all reward in one recorded set.
- 748 failed runs declared completion within ten turns.
- Every failure received less reward than every success.
- Hidden tests checked the machine’s final state.
Scores improved across suites
Reported averages rose on tested benchmarks, though results varied by suite.
| Benchmark | Base | Final |
|---|---|---|
| Terminal-Bench 4.0 | 0% | 19.84% |
| Terminal-Bench 3.0 | 0% | 16.95% |
| Terminal-Bench 2.1 | 51.70% | 86.52% |
| Terminal-Bench Hard | 31% | 79% |
What remains uncertain
The reported gains are benchmark measurements, with open questions about broader use.
- Long-horizon performance remained limited: five of 46 tasks on average.
- A verifier can pass tests without capturing every part of a request.
- Transfer to everyday software work, independent replication and training cost remain unclear.
Why Verifying Completion Matters
The paper’s practical contribution centers on what counts as success. Terminal agents work in a command line, where they can inspect files, run programs and make changes. They can stop before finishing, or say a task is complete when it is not. Terminal-RSI uses hidden tests that inspect the machine’s final state, so an agent’s own declaration does not determine whether it succeeded.
The authors report that, in one set of recorded training attempts, 748 failed runs declared completion within ten turns. They also describe a change to the reward design: under the earlier reward, failed attempts received 50.09% of all reward; the revised scheme assigns every failure less reward than every success. That addresses a specific training problem, but it does not establish that the agent reliably completes arbitrary real-world tasks.
For developers, the distinction has direct relevance: a system that reports progress or passes some checks may still have failed the requested task. The study’s results suggest that verified outcomes and carefully designed rewards can affect benchmark performance. Whether those methods transfer to other systems and less controlled work remains open.
Products Worth Considering
LG UltraGear 27-inch QHD OLED Gaming Monitor, 240Hz
Logitech MX Master 4 for Mac Wireless Ergonomic Mouse, White Silver
Apple AirPods Pro 3 Wireless Earbuds with Active Noise Cancellation
As an affiliate, we earn on qualifying purchases.
How the Five Stages Work
Terminal-RSI uses a combination of supervised fine-tuning and reinforcement learning. In the supervised stages, the model learns from examples derived from its own attempts, but only after a verifier accepts the result and a rewritten plan passes again. The reinforcement-learning stages score new attempts and update the model toward higher-scoring behavior. The authors also carry the critic network between stages and recalibrate it before each refinement phase.
A control reported in the preprint helps explain the ordering. Reinforcement learning applied directly to the base model stayed at 0% on Terminal-Bench 4.0 at the checkpoints the authors recorded. When the self-rewrite stage came first, the following reinforcement-learning stage reached 16.67%. This comparison supports the authors’ staged approach on that benchmark; it does not show that the same sequence will work across other tasks or models.
The benchmark results vary by suite. The base model scored 51.70% on Terminal-Bench 2.1 and 31% on Terminal-Bench Hard; the final model scored 86.52% and 79%, respectively. On Terminal-Bench 3.0, scores rose from 0% to 16.95%. A separate held-out measure, SWR100, moved from 3% for the base model to 6% and stayed flat from the third stage onward, according to the source article’s account of the paper.
“The most interesting part is not the “self” in the title. It is how the researchers stopped rewarding the agent for looking busy and started rewarding it for actually finishing.”
— Thorsten Meyer AI, summarizing the paper
Limits of the Reported Results
The preprint reports benchmark scores but does not, in the supplied material, establish how well the gains transfer to everyday software work or tasks outside the tested suites. The long-horizon result remains limited: the final model completed an average of 5 of 46 tasks across three runs. The source also notes a case where the final model passed tests without building what the task asked for, underscoring that even a verifier may not capture every aspect of a request.
The findings come from a preprint, and the supplied account does not identify peer review or independent replication. It is also unclear from the source material how costly the full training pipeline was, how the results compare with other current terminal agents under the same conditions, or whether performance would hold across different environments. The paper’s reported gains should be read within those limits.
What Further Evaluation Could Show
The immediate next step is scrutiny of the preprint’s methods and results, including independent attempts to reproduce the benchmark scores. Further evaluations could test whether the reward design and staged training approach work on additional task suites, and whether they improve performance on longer tasks without creating new failure patterns.
The source material gives no announced schedule for a peer-reviewed publication, broader release or follow-up study. Until such details are available, the confirmed development is a preprint reporting gains under a human-designed training process. It does not establish autonomous, human-free recursive self-improvement.
Key Questions
Did Terminal-RSI improve itself without human involvement?
No. The preprint describes five stages designed and run by people. Researchers chose training settings and checkpoints; the model contributed attempts that were filtered and used as training examples.
What score did the final model reach on Terminal-Bench 4.0?
The paper reports 19.84%, up from 0% for the base model. Each score is the mean of three evaluation runs.
What does the self-rewrite stage do?
It takes attempts that pass a verifier, turns them into step-by-step plans, screens those plans, and runs them again from scratch. Plans that pass again can be used as training examples.
Did the agent reliably complete long tasks?
The reported long-horizon result was an average of 5 completed tasks out of 46 for the final model, compared with zero for the base model. That shows a measured gain on the tested benchmark, while leaving substantial room for failure.
Source: Thorsten Meyer AI
