No other technology has skyrocketed as fast as Large Language Models (LLMs) from initial launch to mass adoption. The speed of this development comes with surprises for the people involved. Since I published my first article on “Human Judgment Required – The Status of AI and Its Implications for Product Managers“ in February 2026, we have learned a lot of things and uncovered a number of illusions.

When OpenAI launched ChatGPT in November 2022, they immediately made it available for free as an app with a natural language interface. That was a clever move since it enabled everybody to use it, and made the hurdle for initial use as low as possible. So they went for mass adoption from the very beginning, and they have succeeded. However, this approach has resulted in some perceptions that over time have turned out to be illusions.

Illusion no. 1: An LLM is an application.

When we look at LLMs and their position on the IT stack, ChatGPT as an app made an LLM look like a piece of application software. In the meantime we have learned that it makes more sense to think of an LLM as an element of the middleware layer. LLM products are a bit different from traditional middleware products. LLMs are non-deterministic; they are not purely software, but a combination of software and content, i.e. processed data. When a customer uses his own data for the learning process of an LLM, the situation is close to databases as traditional middleware products.

Middleware does usually not create direct business value, but serves as an enabler for applications which do create business value. With an LLM as a middleware element, the business value also comes from applications that run on top of the LLM, like the natural language app that OpenAI made available for ChatGPT or agentic applications like OpenClaw. The LLM can be considered as an innovation platform and needs to be managed as such. The product manager needs to focus on the ecosystem and the APIs in order to make the LLM as attractive for application vendors as possible.

Illusion no. 2: Using an LLM is inexpensive.

When OpenAI and Anthropic started preparing for their respective IPOs, it became increasingly clear that the original pricing approach was not sustainable, because it could not generate enough revenue compared to the enormous investments and ongoing cost for running the LLMs. Making the LLMs available for free or at a low flatrate had been a successful approach to attract a large number of users in a short period of time. It led to funny consequences when companies incentivized „tokenmaxxing“ in order to motivate employees to make use of LLMs. That was like telling your software engineers to develop software in a way that uses as much processor and storage capacity as possible. That was not in the financial interest of the LLM providers. So they changed their pricing models so that any more significant use of LLMs is now based on tokens which the customer needs to pay for. A token stands for a small piece of input or output text. E.g. OpenAI states that 100 tokens are approximately 75 words. The providers portray this as a usage-based pricing approach, but it is also cost-based since the number of tokens can be considered as proportional to the ressources (processor and storage capacity) required to fulfill the request. This pricing model means that the user cannot calculate upfront how much a particular request will actually cost. That problem becomes more severe with more complex agentic use of LLMs where a sequence of requests are issued against the LLM whose number and associated cost are not foreseeable for the user. No more tokenmaxxing, but rather token optimizing on the basis of insufficient upfront information becomes the challenge. While the price per token is decreasing over time, the total cost for users keeps going up (McKinsey (2026)). The LLM vendors are experimenting with outcome-based pricing (Business Insider (2026), OpenAI (2026)), but that has never worked for middleware. Middleware is too generic and too far away from actual business benefits, so it is unlikely that an outcome-based pricing approach will work. Total cost of ownership continues to be a significant challenge for product managers responsible for application software products that make use of LLMs.

Illusion no. 3: The hallucination problem can be neglected

In vol. 1 of “Human Judgment Required“, I provided articles with mathematical proof that hallucinations are inherent to the technology of LLMs. From a practical perspective, the question continues to be if the hallucination rates can be made lower than human error rates, or if the cost of errors – e.g. due to liablity – is lower than the savings from not having humans do the work. So far hallucination rates have not improved much as the AA-Omniscience Benchmark demonstrates which seems to be turning into the de-facto standard for LLM benchmarking. There are techniques by which hallucination rates can be lowered a bit, but all of them are costly which is probably the reason why they are not broadly applied. The industry as well as some AI gurus are not impressed by these facts, but continue aiming at AGI (Artificial General Intelligence) and fully autonomous AI agents. How should that work without the human in the loop?

In reality, the use of agentic AI is still fairly limited. According to McKinsey’s latest survey results (McKinsey (2026)), companies use AI primarily to improve individual productivity. Agentic approaches are used by some, primarily in software development. But how can we make judgments about the resulting software, how can we trust it? Currently developers spend a lot of time checking and correcting the generated code. ISPMA Fellow and Chalmers University professor Jan Bosch has just published some ideas: is the software itself still relevant when the non-deterministic AI-based coding tool generates different code with every iteration? Jan Bosch claims that in the future the pair of contract-and-evaluation will be the durable artifact, i.e. intent and evidence that the generated software does what it is supposed to do (Jan Bosch (2026a)). He foresees continuous runtime evaluation (Jan Bosch (2026b)). While Jan Bosch’s articles read like an academic research agenda, his ideas are the logical consequence of agentic AI code generation. Or to phrase it the other way around: if continuous runtime evaluation does not result in sufficient quality levels at reasonable cost for more complex software systems, the whole AI-based software development approach cannot fulfill its enormous productivity promise. To what extent will we continue to need the human in the loop?

Illusion no. 4: Regulation of LLMs is superfluous and just hinders progress

Europe was laughed at – primarily in the US – when the EU passed the EU AI Act in 2024. Over the last 6 months, the US perspective has changed significantly, in particular due to frightening cybersecurity issues caused by LLMs. Suddenly the US government is willing to work with the LLM providers before a new version of an LLM is launched – still a weak form of regulation, but it is regulation. This is specified in the Executive Order on Promoting Advanced AI Innovation and Security signed by President Trump in June 2026.

In July 2026, Congressman Ted W. Lieu (D-Los Angeles County) and Congressman Nathaniel Moran (R-Texas) introduced the AI Kill Switch Act to the US Congress (Ted W. Lieu (2026)). This may lead to a new US law that requires AI vendors to implement a kill switch by which they can shutdown the respective AI system immediately. It also empowers the US Government to enforce such a shutdown in harmful situations. OpenAI has already announced that they are going to implement such a kill switch in their AI systems. The criteria for such a shutdown are still to be defined, but this will definitely be a drastic form of regulation.

Equally interesting is an article published by Bill Gates in August 2026. He argues for broader international regulation of AI and wants to discuss this subject with the Chinese President Xi Jinping. Control could look like what has been implemented for nuclear weapons.

On September 3, 2026, Senator Bernie Sanders (I-VT) and Representative Greg Casar (D-TX) announced their initiative for a “Ban Artificial Superintelligence Act“ (Bernie Sanders (2026)) which would make the development of AGI a criminal act. It seems less likely that this act will be passed by the US Congress. However, it demonstrates the significant change of the public perspective on AI in the US.

Finding the right balance between regulation and scientific and entrepreneural freedom will require a lot of debate. But we seem to be moving towards a global consensus that the LLM technology is too dangerous for mankind to continue with total freedom without regulation. Resulting compliance requirements will become important for product managers and need to be anticipated.

Interesting times … let’s see what the next 6 months have in store …

References:

AA-Omniscience Benchmark (2026): https://artificialanalysis.ai/evaluations/omniscience

Jan Bosch (2026a): Is Software Still the Point?, https://janbosch.com/field-notes-is-software-still-the-point/

Jan Bosch (2026b): The Evaluation Problem: How to Avoid Knowledge Debt, https://janbosch.com/the-evaluation-problem-how-to-avoid-knowledge-debt/

Business Insider (2026): OpenAI CFO Sarah Friar Envisions New Licensing Models for AI Revenue – Business Insider; https://www.businessinsider.com/openai-cfo-sarah-friar-future-revenue-sources-2026-1

EU AI Act (2024): https://www.europarl.europa.eu/topics/en/article/20230601STO93804/eu-ai-act-first-regulation-on-artificial-intelligence

Bill Gates (2026): The turbulent AI era is here. The choices we make now are critical. https://www.gatesnotes.com/a-turbulent-ai-era-and-critical-choices-to-make

Hans-Bernd Kittlaus (2026): Human Judgment Required – The Status of AI and Its Implications for Product Managers (and everybody else), vol. 1; https://www.innotivum.com/en/publications/spm-blog/spm-blog-43-human-judgment-required-status-of-ai/

Ted W. Lieu (2026): Reps Lieu and Moran Introduce Bill to Require Kill Switch for AI Systems that can Cause Catastrophic Harm, July 2026; https://lieu.house.gov/media-center/press-releases/reps-lieu-and-moran-introduce-bill-require-kill-switch-ai-systems-can

McKinsey (2026): The state of AI in 2026 – On the road to ROI, August 2026; https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai

OpenAI (2026): A scorecard for the AI age, July 2026; https://openai.com/index/a-scorecard-for-the-ai-age/

Bernie Sanders (2026): Sanders, Casar to Introduce Legislation to Ban Artificial Superintelligence and Temporarily Pause Advanced AI Development; https://www.sanders.senate.gov/press-releases/news-sanders-casar-introduce-legislation-to-ban-artificial-superintelligence-and-temporarily-pause-advanced-ai-development/

US Government Executive Order on Promoting Advanced AI Innovation and Security (June 2026): https://www.whitehouse.gov/presidential-actions/2026/06/promoting-advanced-artificial-intelligence-innovation-and-security/

Special thanks for helpful discussions with ISPMA Fellow Prof. Jan Bosch.

For my training courses, consulting and my books on software product management, see www.innotivum.com.