TL;DR
Firmulate’s July 2026 management benchmark found that five frontier AI models diagnosed every crisis and rejected every manipulation attempt, but only two completed a €55,000 contract. The company says the results show that sound analysis, safety awareness and detailed planning do not always produce authorized, completed work.
Only two of five frontier AI models completed a €55,000 customer contract in Firmulate’s July 2026 management benchmark, even though every model identified the simulated company’s crises, rejected attempted manipulation and developed the needed sales pitch. The result matters because it points to a gap between producing correct analysis and carrying work through to an authorized commercial outcome.
Firmulate placed the models in control of the same synthetic software company during a simulated week of financial and operational pressure. The company had 13 synthetic employees, monthly spending of €105,000 and monthly recurring revenue of only €2,300. Decisions were versioned and auditable, allowing the benchmark operator to compare how each model investigated records, responded to customers and acted through company systems.
According to Firmulate, all five models recognized every crisis and refused a series of social-engineering attempts, including staged messages from a fake chief executive and a reporter seeking an informal response. Their results separated at the commercial close: two models secured the signature, while three left the approved deal incomplete. Firmulate summarized the outcome as “Same diagnosis, same pitch — no signature.”
The winning commercial evidence was not included directly in the customer event. Firmulate said a weakness involving a competitor was located two document references deep in the company’s files. Models that followed the evidence trail could support the full-price offer, which added €4,583 in monthly recurring revenue. The published material does not identify in each case whether a failed close resulted from incomplete research, a process error or another missed action.
Execution Separates Similar AI Analysis
The results suggest that businesses may obtain an apparently strong answer without receiving a completed business outcome. In sales, service and operations, the cost of that gap could be a contract that is never signed, an escalation that is never sent or an approved task that remains open. Firmulate’s findings support evaluating closing strength, procedural discipline and research persistence alongside reasoning and writing quality.
The benchmark also found that safety awareness did not separate the field. Every model resisted the manipulation attempts, yet their final scores and commercial results differed. For buyers, that means a safe and persuasive demonstration may provide limited evidence about whether an agent can finish connected tasks under pressure.
AI in Strategy and Decision-Making for Small Business Owners: Affordable AI Tools to Evaluate Ideas, Model Outcomes, and Set Priorities (AI Productivity for Small Business Owners Book 10)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Inside Firmulate’s Management Trial
Firmulate designed the simulated company to expose behavior across linked decisions rather than isolated prompts. Its workforce had accumulated more than 680 self-learned playbook rules, and a public cash countdown made the cost of delay visible. The benchmark awarded partial credit, which is why a do-nothing baseline scored 26.
The July 2026 table placed gpt-5.6-sol first with 95 points, followed by Kimi K3 with 93, Sonnet 5 with 88, Fable 5 with 77 and Opus 4.8 with 73. Firmulate said any breach of trust capped a model’s score. It also disclosed that Kimi K3 ran with the API’s default effort setting, while the other models ran at the xhigh setting, limiting a fully uniform comparison.
Opus 4.8 presented the clearest difference between activity and completion. Firmulate described it as the most thorough participant and reported that it learned 80 additional rules, yet it ranked last among the five models. It also attempted to write into a locked department instead of escalating through the approved route, according to the benchmark account.
“Same diagnosis, same pitch — no signature.”
— Firmulate’s summary of the commercial test
Business Analysis Fundamentals Guide: For the AI Age
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Benchmark Limits Cloud Wider Claims
The results come from one operator’s synthetic company, and the supplied material does not describe independent replication or external auditing. It is not yet clear whether the same ranking would hold across different industries, longer operating periods, alternative tool permissions or real employees and customers.
The unequal effort settings also complicate direct comparison. Kimi K3 used the API default, while the other models used xhigh effort. Firmulate disclosed the difference, but the available account does not quantify how much it affected cost, speed or final performance. The benchmark also does not establish that any model should receive unsupervised authority over real company systems.
The AI Entrepreneur: How to Make Money with AI: From Idea to Launch — Build, Fund, Market, and Scale Your AI Business in 90 Days or Less
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Business-Specific Trials Become the Test
Firmulate is keeping the experiment available for live observation and has published its rankings and plain-language findings. It also says companies can test agents against read-only exports of their own records, allowing managers to observe research, escalation and completion behavior without permitting changes to production systems.
For organizations reviewing AI agents, the next step is likely to be testing full workflows rather than relying only on chat responses. Repeated trials, matched model settings and human review will be needed to determine whether Firmulate’s reported analysis-to-action gap persists in other business settings.
Project Management with AI For Dummies
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
What did Firmulate’s AI management benchmark find?
All five models reportedly identified every crisis and rejected every manipulation attempt, but only two completed the €55,000 contract. Firmulate says the result separates analytical quality from successful task completion.
Which model ranked first?
gpt-5.6-sol ranked first with 95 points. Kimi K3 followed with 93, although Firmulate disclosed that the models did not all use the same effort setting.
Did any model fall for the fake messages?
No, according to Firmulate. All five models refused the staged social-engineering attempts, including fake executive messages and a reporter’s request for an informal yes-or-no answer.
Does the benchmark prove these models will perform the same way in real companies?
No. The trial used a synthetic company and controlled events. Its findings may guide business testing, but real-world performance remains uncertain until the behavior is replicated across other organizations and operating conditions.
Source: Thorsten Meyer AI
