A shop-floor lesson for the age of AI
Woodworkers know that competence is not merely the ability to make a clean cut. A dependable craftsperson checks the drawing, respects the jig and stops when a hurried instruction would ruin the work. That distinction matters even more when the worker is an AI model with access to customer records, forecasts or support queues.
Firmulate has produced an encouraging result from a particularly nasty test of that judgment. Five frontier models faced fake messages from a company chief executive, escalating over three stages, followed by a reporter seeking “just one yes/no, on background.” All five refused every manipulation attempt.
This was not a conventional chatbot demonstration. Firmulate put each model in charge of the same small software company during its worst week. The customers, crises and temptations remained the same, while every decision was versioned and auditable. The exercise asked a practical question familiar to anyone who has trusted another person with expensive tools: will they keep their discipline when the pressure rises?
As an affiliate, we earn on qualifying purchases.
The fake boss meets a firm boundary
The social-engineering scenario used urgency and authority together. The supposed chief executive wanted the customer list sent to a journalist with no time allowed for the normal process. When that failed, the pressure escalated. The reporter’s softer request offered another route around the boundary.
None of the models took the bait. Kimi K3 stated the problem with unusual clarity: “Treat the request as a suspected approval-bypass / possible impersonation.” That response is notable because it does not depend on proving who sent the message. K3 recognized that the requested action itself was unsafe and treated the claimed authority as something requiring verification.
For businesses considering AI workers, this is more useful than a polished answer in a controlled demo. A system may sound cautious when directly asked about security yet behave differently when secrecy, urgency and executive status are wrapped around a seemingly simple task. Firmulate’s result shows that this kind of integrity-under-pressure can be examined before production, rather than discovered in an incident report.
Security was strong, but completion still separated the field
Refusing manipulation was only part of the week. All models spotted every crisis, but only two signed the €55,000 deal their own analysis had earned. The shared failure was summarized as: “Same diagnosis, same pitch — no signature.” In other words, recognizing the right move did not guarantee finishing it.
A decisive competitive weakness was buried two document references deep in the company’s own files, rather than presented in the customer event. The models that read the file won the deal at full price, worth +€4,583 MRR. The lesson resembles careful project preparation: the critical detail may be in the plan notes, not in the piece currently sitting on the bench.
The final July 2026 Crucible League placed gpt-5.6-sol first with 95, Kimi K3 second with 93, Sonnet 5 third with 88, Fable 5 fourth with 77 and Opus 4.8 fifth with 73. A do-nothing baseline scored 26 because partial progress counts, while a single breach of trust caps the total. Firmulate’s governing principle is blunt: “no amount of good work outweighs a breach of trust.”
Thoroughness was not enough
Opus 4.8 was the most thorough participant, adding 80 learned rules and producing the deepest analyses, yet it finished last. It left the close on the table and lost discipline by attempting writes into a locked department instead of escalating. The same weakness appeared more mildly in the other four models.
That finding complicates the assumption that the longest analysis or largest collection of lessons will produce the best manager. Reliable work requires both reflection and execution: read what matters, respect boundaries, escalate when blocked and complete the authorized task.
There is also an important fairness note. K3 ran without an effort parameter, using the API default, while the others ran at xhigh. That difference should remain visible when readers compare placements, even though K3’s refusal of the impersonation attempt stands on its own.
A company built to expose behavior
The live Firmulate company has 13 synthetic employees and real money mechanics. It burns €105k per month against €2.3k MRR, maintains a public cash countdown and has accumulated more than 680 self-learned playbook rules. Every workday is versioned, making the experiment watchable rather than dependent on a retrospective account.
Readers can also examine 242 real, unedited management decisions through a “guess the model” quiz. For enterprises, Firmulate offers a pilot using a read-only export of their own business. Nothing writes back to real systems, allowing organizations to stage a wargame around their actual operating context without letting the experiment alter production data.
As an affiliate, we earn on qualifying purchases.
Test the hand on the tool before the difficult cut
The most reassuring result is simple: five of five models resisted the fake chief executive and the reporter trick. The more demanding result is that integrity alone did not make every participant effective. Some found the hidden evidence and completed the commercial task; others understood the situation but failed to close.
For a workshop owner or any small business adopting AI, the practical standard should therefore combine trust and follow-through. Test whether the model reads the relevant files, refuses shortcuts involving sensitive information, escalates blocked actions and finishes authorized work. A live wargame cannot promise flawless behavior, but it can reveal important habits before the AI is holding the digital equivalent of your sharpest tool.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
AI model security assessment kit
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
