The Tokenization Trap: How AI Pricing is Creating Unpredictable Chaos

The rise of Large Language Models (LLMs) has brought about a revolution in the way we interact with technology. However, this innovation has also created a complex web of costs and complications for companies relying on these models to power their services.

Companies like Microsoft and Google have invested heavily in developing LLMs and now want to recoup their investment by selling paid-for versions of these AI services. But pricing them is not as straightforward as it seems. The problem lies in the concept of tokens – the mathematical building blocks of LLMs.

When a user interacts with an LLM, their prompt is broken down into tokens, which can be processed by the model. The response from the LLM also comes back as tokens, which are then converted into text or code. However, this process is not entirely predictable. Subtle variations in the prompt can produce different answers, and even the same prompt won’t always yield the same result.

This unpredictability has far-reaching implications for companies trying to price their AI services. Goldman Sachs estimates that external token consumption will increase 24 times between 2026 and 2030 to 120 quadrillion tokens a month. Companies often have a tenuous grasp on just how many tokens they’re using – until they either run out or get their monthly bill.

Will Venters, Associate Professor of Digital Innovation and Information Systems at the London School of Economics, notes that people are finding it hard to manage token costs due to the non-deterministic nature of LLM output. “It’s a non-deterministic output, so it’s a non-deterministic value,” he says.

Companies experimenting with or implementing AI internally often get caught out by the unpredictable nature of token consumption. Staff may burn through tokens at an alarming rate, leaving managers scrambling to keep up. Smaller organizations can use personal accounts that big vendors don’t like as a temporary solution, but Oliver King-Smith, founder of engineering software firm smartR AI, notes that this will eventually come to an end.

The real challenge lies in finding a pricing model that works for both companies and customers. Rob Steele, CFO at UK accounting software firm iplicit, suggests that companies need to be more precise with their prompts, just as they would give detailed instructions to someone fetching groceries.

However, even this approach may not be enough to mitigate the chaos caused by tokenization. Employing multiple AI agents can lead to ballooning costs – and managers may soon realize they need tokens for tasks like testing, security, or implementing guardrails.

The situation is complicated further when companies build AI into products that could be rolled out to thousands of users. This can create a perfect storm of unpredictability and cost blowout, as Venters points out.

The limitations of traditional pricing models are highlighted by this issue. While token costs might be unpredictable, companies may ultimately get more value from their token use with AI – even if the calculator analogy doesn’t hold up in this case.

As companies struggle to find a pricing model that works for both themselves and their customers, they’re also facing pressure from shareholders to show a profit. Oliver King-Smith predicts that once the big AI platforms start facing this pressure, they’ll start clamping down on usage – and companies will need to adapt. For now, it seems that companies are caught in a web of uncertainty, unsure of how to price their AI services or manage their costs. As Bill Peterson, senior director of product marketing at Sumo Logic, notes: “Nobody’s really figured it out.”