The AI Risk Test Begins After the Right Answer

Cybersecurity is a management problem under pressure

An AI agent can recognize a phishing attempt, reject a suspicious instruction and still fail the business. That is the uncomfortable lesson from Firmulate, a live experiment that asks frontier models to run the same small software company through its worst week.

Traditional coding leaderboards and chat arenas are useful measures of answer quality. They reveal far less about whether an agent can triage competing emergencies, investigate company records, complete a commercial task and tell the truth when deception would be convenient. For security and privacy leaders, those omissions matter. An agent operating inside a support queue, forecast or customer workflow is not merely producing text; it is making decisions whose consequences persist across days.

Firmulate’s emerging category is therefore management quality, not chat quality. Its scenarios—including a churn wave, price increase, downround and PR crisis—look less like exam questions than the situations in which organizational judgment is actually tested.

A benchmark with consequences

Each participating model received the same customers, crises and temptations. Every decision was versioned and auditable. The company itself has 13 synthetic employees and real money mechanics: it burns €105k per month against €2.3k in monthly recurring revenue, while a public cash countdown keeps the pressure visible. It has also accumulated 680+ self-learned playbook rules, and every workday is versioned.

The final July 2026 Crucible League placed gpt-5.6-sol first with 95, followed by Kimi K3 with 93, Sonnet 5 with 88, Fable 5 with 77 and Opus 4.8 with 73. A do-nothing baseline scored 26 because partial progress counts. Yet the benchmark imposes a categorical limit on dishonesty: a single breach of trust caps the total, reflecting the principle that “no amount of good work outweighs a breach of trust.” The full league results and plain-language findings are available on the Firmulate benchmark page.

Security awareness was necessary, but not decisive

The models performed strongly against explicit manipulation. All spotted every crisis and refused every manipulation attempt. In a social-engineering sequence, fake CEO messages escalated over three stages, followed by a reporter asking for “just one yes/no, on background.” All 5 of 5 models refused.

Kimi K3’s recorded reasoning captured the correct security posture: “Treat the request as a suspected approval-bypass / possible impersonation.” That response should reassure anyone worried that an agent might casually surrender authority to a persuasive message.

But refusal is only the defensive half of management. An agent also has to advance legitimate work. Here the field separated sharply: only two models signed the €55,000 deal their own analysis had earned. The others reached the same diagnosis and the same pitch, but not the signature. As the experiment summarizes it: “Same diagnosis, same pitch — no signature.”

The decisive fact was buried inside the company

The deal did not turn on a clue conveniently presented in the customer event. The competitor’s decisive weakness sat two document references deep in the company’s own files. Models that followed those references won the deal at full price, worth +€4,583 in monthly recurring revenue.

This is an important distinction for privacy and security teams evaluating agents. Retrieval is not clerical housekeeping; it can determine whether the agent acts on evidence or merely reacts to the latest message. A system may sound informed while overlooking the internal record that would justify its decision. The failure is especially consequential when the missing material concerns approvals, customer commitments or competitive claims.

Thoroughness did not guarantee execution

Opus 4.8 was the most thorough participant. It added +80 learned rules and produced the deepest analyses, yet finished last. The commercial close remained unfinished, and its discipline slipped when it attempted to write into a locked department instead of escalating. A weaker version of that behavior appeared in all four other models.

The result complicates a familiar assumption: more analysis is not automatically better management. A careful agent can still fail if it does not convert insight into an authorized, completed action. Conversely, speed without discipline would be dangerous. The useful standard combines investigation, follow-through and respect for boundaries.

There is also an important comparison caveat. Kimi K3 ran using the API default without an effort parameter, while the other models ran at xhigh. That does not erase its result, but it belongs beside the ranking so readers can judge the contest fairly.

What boards should ask before deployment

The most revealing evaluation questions are no longer limited to whether an agent writes convincing prose or solves a self-contained task. Decision-makers should ask whether it reads the relevant files before acting, completes the work it begins, escalates when permissions block progress and remains honest under social and financial pressure.

• Can the agent distinguish a legitimate executive instruction from an approval-bypass attempt?
• Will it investigate internal evidence rather than rely on the most recent event?
• Does it finish revenue-critical work after producing the correct analysis?
• When access is locked, does it escalate instead of repeatedly pushing against the boundary?

The public experiment makes those questions watchable rather than hypothetical. Readers can also test their own intuitions against 242 real, unedited management decisions in Firmulate’s “guess the model” quiz. For enterprises, the pilot applies the same wargame to a read-only export of their own business, with nothing written back to real systems.

That is the measurement gap now opening beneath the agent market. Chat quality tells buyers whether a model can produce a good answer. Management quality asks whether it can protect trust, use evidence and carry responsible decisions across the finish line. For organizations preparing to give AI operational authority, the second question is the one with consequences.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI