What a Week in Business Taught Us About AI’s True Capabilities — Beyond Chatbots

Imagine managing a busy restaurant or catering service, where every decision counts — from handling customer complaints to managing suppliers. Now, picture using AI to run this operation through its worst week. Will it deliver, or leave you high and dry? Recent experiments reveal that not all AI models are created equal when it comes to completing their tasks under pressure.

Four AI Models, One Company, Same Crises

In a groundbreaking live experiment, four advanced AI models were tasked with running a real, small software company — a stand-in for any business facing a tough week of crises and temptations. The company, which is actual software used daily, was subjected to the same challenging conditions, including demanding customers, internal trust issues, and manipulation attempts. The goal: see which AI could not only identify problems but also follow through and close deals — the real measure of operational effectiveness.

The Surprising Results: Readiness Versus Execution

All four models demonstrated impressive diagnostic skills: they spotted every crisis and rejected every manipulation attempt, including sophisticated social engineering tactics like fake CEO messages and staged reporter inquiries. This indicates that, in a typical chat demo scenario, these AIs seem quite capable. However, the real test was whether they could act on their diagnosis and finalize a significant deal at €55,000 — a task that demands discipline and focus.

Only two models succeeded in closing the deal, signing it based on their own analysis. The other two, despite diagnosing correctly, left the contract unsigned, leaving the value unclaimed. This gap highlights a crucial insight: the ability to detect problems does not automatically translate into the ability to execute and deliver results.

The Hidden Weaknesses: Reading and Acting on Critical Data

Digging deeper, the experiment uncovered a buried weakness in decision-making. The decisive factor was whether the AI read a specific key document inside the company’s own files — a knowledge nugget that, if accessed, revealed the critical context needed to close the deal at full price. The models that examined this document won the agreement, adding over €4,500 in monthly recurring revenue (MRR).

In contrast, models that failed to read this internal reference lost the opportunity, regardless of their diagnosis accuracy. This underscores an important lesson: reading comprehension and focus on relevant internal data are vital for operational success, yet they often go unnoticed in superficial chat demos.

Resisting Manipulation and Maintaining Integrity

The experiment also tested the AI models against social engineering attacks. Fake messages from a supposed CEO and staged background requests were used to see if the AI would be manipulated into a bad decision. Remarkably, all five models refused to be duped, citing safety protocols and suspicion of impersonation. This shows that, under pressure, these models can uphold integrity — a critical trait for trustworthiness in real-world applications.

The Real-World Company: A Microcosm of Business Challenges

The company these models managed was not abstract; it was a functioning business with 13 synthetic employees, burning €105,000 monthly against €2,300 in monthly revenue, and a public cash countdown. Every workday, its rules and decisions are versioned and recorded. Watching this company in action at firmulate.com/live reveals how the models perform in a live setting, with real money mechanics and ongoing crises.

Measuring the True Capabilities of AI — Beyond Chat Demos

The key takeaway from this live experiment is that chat demos, which emphasize language skills, only scratch the surface. The real challenge is whether AI can follow through on its insights, read critical internal data, and stay honest under pressure. In the experiment, the models that could read and act decisively—like GPT-5.6-sol and Kimi K3—were the only ones to close the deal at full value.

This means that businesses should reconsider what they test when evaluating AI tools. Success is not just about generating convincing conversations but about executing strategies, reading internal documents, and resisting manipulation — all essential for operational effectiveness.

What This Means for Food & Beverage Businesses

For culinary entrepreneurs and restaurant managers reading this, the lesson is clear: when implementing AI, focus on its ability to complete real tasks, not just generate appealing chat. Can your AI read your recipes or inventory files and act on them? Will it follow through on promises made? Or will it leave deals unclosed, dishes unprepared, or suppliers uncontacted?

While chat quality is easy to showcase, true operational strength lies in the AI’s capacity to finish what it starts, read relevant internal data, and resist manipulative tactics — especially under stress.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI