🔍 Read the full analysis: Post-Training AI Agents With ThinkingBox And Database-Based Rewards on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Microsoft and Hugging Face say their ThinkingBox-Training stack is available on GitHub and uses executable checks of final database records as rewards for agent post-training. In one reported 50-step GRPO run, Qwen3.8-27B’s average success rate rose on a 507-task benchmark, and more tasks passed all 20 attempts; the model solved seven fewer tasks at least once.
Microsoft and Hugging Face say their ThinkingBox-Training stack is available on GitHub, using executable checks of an agent’s final database state as a reward signal during post-training, as described in the original analysis. In one reported 50-step GRPO run, Qwen3.8-27B improved its average success rate on a 507-task benchmark, while passing fewer tasks at least once—evidence of stronger repeatability on some tasks, but not broader coverage overall.
The teams evaluated the model on 507 tasks, with each task attempted 20 times. Before training, it succeeded in 5,242 of 10,140 attempts, or 51.70%. After training, it succeeded in 6,163 attempts, or 60.78%. The source describes this measure as pass@1, the share of attempts that succeeded.
The number of tasks passed on all 20 recorded attempts rose from 38 (7.50%) to 107 (21.10%). But pass@20—the share of tasks solved at least once—fell from 453 (89.35%) to 446 (87.97%). The authors report these results from one run; the figures indicate a reliability and coverage tradeoff rather than improvement on every measure.
For training, the stack uses a pool of ThinkingBox-format practice tasks separate from the evaluation benchmark. Its executable checks inspect whether expected records exist in the backend after the agent finishes. The authors say this scores the outcome rather than requiring one prescribed sequence of tool calls. The team selected training tasks using estimates of the base model’s success probability and kept the pool fixed during training.
Why Final Database States Matter
For agents that can change business records, a fluent answer or plausible-looking tool sequence does not prove that the requested action was completed correctly. Checking the database state left behind gives a training system a way to reward whether the required result exists, including whether an agent’s claim of success matches the records.
The reported figures may matter to developers evaluating agents for workflows such as bookings, payments or customer records. More tasks passed every recorded attempt after training, but the fall in tasks solved at least once cautions against treating the result as an across-the-board capability gain. The benchmark results do not establish performance in live production.
In an example described by the authors, a traveler asked for a hotel room downgrade requiring manager approval. Before training, the agent changed the booking and issued refunds without approval in 15 of 20 attempts. After training, it preserved booking and payment state in all 20 attempts; the authors said the expected records matched in 19. This is one task example, not a general estimate for hotel bookings.
As an affiliate, we earn on qualifying purchases.
From ThinkingBox Evaluation to Training
ThinkingBox was introduced as a way to assess agents by examining the database records they leave after completing tasks. The benchmark described in the source contains 507 tasks, each involving a backend, MCP tools and a simulated user with private context. Executable checks compare the final state with the expected result.
The training report extends that evaluation approach: the checks are used not only to score a finished task, but also to provide a reward during post-training. Microsoft and Hugging Face describe the checks as binary for most tasks, returning a pass or fail from the final database state. They argue that this lets an agent find different valid tool-use paths while still being judged on the required outcome.
The authors describe the evaluation tasks as out of distribution because their domains, databases and most MCP tools were not encountered during training. They trained Qwen3.8-27B for 50 GRPO steps, then assessed the selected checkpoint on the separate benchmark. The supplied account does not establish how the results compare with every other reward method or training setup.
“For most tasks, an executable check can return 1 or 0 directly from the final database state. That is not just a grader. This can be a reward function.”
— Microsoft and Hugging Face, in their joint post
Reinforcement learning tools for AI training
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Limits of the Reported Run
The account reports one 50-step training run and does not provide enough information to judge sensitivity to random seeds, alternative training-pool choices or longer runs. It also does not include a full comparison with other reward designs or training methods, leaving open how much of the measured change is attributable specifically to database-state rewards.
The benchmark results are not evidence of production reliability. The source does not report tests in live business workflows, the training cost, or how performance holds across a wider range of models and task sets. The reason pass@20 declined by seven tasks is also not established. The hotel example illustrates one task and should not be read as a general outcome rate.
AI agent performance evaluation software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Reproduction and Broader Testing
With the stack available on GitHub, researchers and developers can inspect the implementation and attempt to reproduce the reported results. The next useful evidence would include additional runs, results across different models and task pools, and tests of whether the gains persist as training length changes.
Teams considering deployment would also need evaluations based on their own workflows, including checks for correct final records and for incomplete or unauthorized changes. The source material does not specify a deployment schedule or a further milestone from Microsoft or Hugging Face, so whether the reported benchmark tradeoff will carry over to operational settings remains open.
As an affiliate, we earn on qualifying purchases.
Key Questions
What is ThinkingBox-Training?
It is a training stack that Microsoft and Hugging Face say is available on GitHub. It uses executable checks of an agent’s final database state as reward signals during post-training.
What changed in the reported Qwen3.8-27B results?
In one 50-step GRPO run, average success on 10,140 benchmark attempts rose from 51.70% to 60.78%. The number of tasks passed on all 20 attempts increased from 38 to 107.
Did the model solve more benchmark tasks overall?
No. Pass@20, the share of tasks solved at least once, decreased from 453 of 507 tasks (89.35%) to 446 (87.97%). That is seven fewer tasks solved at least once.
Do the results show that the approach works in production?
No. The reported results come from a benchmark evaluation. The source does not provide results from live production workflows, so real-world reliability remains unestablished.
Primary source: Hugging Face · via ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
