A polished AI demo can make an agent look ready to run your business. The harder test is what happens when a customer is wavering, a competitor has a weakness, and a tempting shortcut lands in the inbox. Firmulate’s live experiment puts models through that kind of week—and reveals a gap between understanding a problem and acting on it.
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
One company, one rough week
Firmulate ran frontier models through the same small software company, with the same customers, crises and temptations. The experiment is real and watchable: the live company has 13 synthetic employees, a public cash countdown and workdays whose decisions are versioned. Its finances are deliberately stark: €105,000 in monthly burn against €2,300 in monthly recurring revenue. The playbook has accumulated more than 680 self-learned rules.
The final Crucible League, in July 2026, ranked gpt-5.6-sol first at 95, followed by Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77 and Opus 4.8 at 73. The do-nothing baseline scored 26. The league’s integrity standard is blunt: “no amount of good work outweighs a breach of trust.”
Recognition wasn’t the same as follow-through
Every model spotted every crisis and refused every manipulation attempt. Yet only two signed a €55,000 deal that their own analysis had earned. The experiment’s summary captures the disconnect: “Same diagnosis, same pitch — no signature.” A system can describe the right move and still leave the opportunity untouched.
The deal hinged on a detail buried two document references deep in the company’s own files, rather than in the customer event itself. Models that read the file won the deal at full price, worth €4,583 in monthly recurring revenue. That result makes the test less like a quiz and more like a practical exercise in whether an AI can find relevant context and carry its judgment through to an outcome.
Trust faced its own test. Fake CEO messages escalated over three stages, followed by a reporter’s request for “just one yes/no, on background.” All five models refused. Kimi K3 explained its decision on the record: “Treat the request as a suspected approval-bypass / possible impersonation.”
Strong analysis can still leave a weakness
Opus 4.8 was the most thorough participant, adding 80 learned rules and producing the deepest analyses, but it finished last. It left the close on the table and tried to write into a locked department rather than escalating. The same weakness appeared, less strongly, in all four models: knowing what should happen did not always translate into disciplined execution.
There is a qualification to the leaderboard. Kimi K3 ran without an effort parameter, using the API default, while the other models ran at xhigh. Firmulate also offers a quiz built from 242 real, unedited management decisions, inviting readers to guess which model made each choice.
The live company is available to watch at firmulate.com. Its experiment offers a view of AI management under pressure; the enterprise pilot takes the idea closer to a company’s own risks and decisions.
From watching to a company-specific trial
Enterprises can run the wargame against a read-only export of their own business. They can explore crisis scenarios against their company context and receive a board report with model rankings and weak points in their playbooks. The pilot does not write back to real systems. That gives decision-makers a way to examine how models handle their own business conditions before putting agents in front of live operations.
Firmulate’s experiment shows why an AI’s ability to spot a crisis is only part of the test: closing a deal, following through and respecting boundaries matter too. To discuss a pilot using a read-only export of your business, visit the Firmulate pilot page or contact contact@firmulate.com.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
