The Token Diet: Building Enterprise AI Without Wasting Intelligence

AI should make organizations more efficient. Its implementation should be efficient too.

Every day, we go deeper into what we call token economics: understanding how to generate more organizational value from AI while consuming fewer computational resources.

This is important for our customers, and it is equally important for us.

The easiest way to build an AI system today is often to send more data to a larger model, provide it with a longer prompt, add several agents, and hope that enough intelligence will eventually produce the desired result.

It may work.

But it is not necessarily good engineering.

At WebTech, we do not want to become another sales channel for tokens on behalf of the global software giants. Our responsibility is not to maximize model usage. It is to create the best possible business outcome with the lowest reasonable level of complexity, cost and resource consumption.

Internally, we have started calling this approach the Token Diet.

Not because tokens are inherently bad. Tokens are the fuel that allows modern language models to understand, reason and generate. But fuel should be consumed where it creates value—not poured indiscriminately into every stage of every process.

Using a frontier model to solve a simple deterministic problem is the AI equivalent of firing a 120-millimeter tank shell at a fly.

The fly may disappear. But that does not make it a sensible solution.

AI Is a Tool, Not the Default Architecture

There is a growing tendency to treat AI not merely as the best way to perform a task, but as the only way.

Need to classify a record? Call a model.

Need to extract a date? Call a model.

Need to determine whether a field is empty? Call a model.

Need to route a process? Create an agent.

Need to call an API? Create another agent to supervise the first one.

This approach may look advanced in a demonstration, but at enterprise scale it can become expensive, unpredictable and difficult to maintain.

Our starting principle is therefore straightforward:

What can be performed reliably and efficiently without a language model should usually be performed without one.

Traditional software has not become obsolete. Rules, SQL queries, regular expressions, search indexes, workflow engines, deterministic APIs, event processors and conventional machine-learning classifiers remain extremely effective.

The engineering challenge is to identify the point at which deterministic computing stops being sufficient—and language-model intelligence becomes genuinely valuable.

That is where the tokens should be deployed.

From AI-First to Intelligence-Appropriate Design

We prefer to think in terms of intelligence-appropriate architecture rather than AI-first architecture.

A well-designed enterprise process might include several layers:

1. Deterministic preprocessing

   
   

   Validate fields, normalize formats, remove duplicates, identify document types and apply known business rules before involving a model.

2. Retrieval and filtering

   
   

   Use metadata, keyword search, database queries, vector retrieval or hybrid search to locate only the information relevant to the current task.

3. Lightweight intelligence

   
   

   Use smaller or specialized models for classification, extraction, routing and other constrained tasks.

4. Advanced model reasoning

   
   

   Invoke a more capable model only when the request requires ambiguity resolution, synthesis, complex reasoning or natural-language generation.

5. Deterministic execution and validation

   
   

   Once the model has produced a decision or structured instruction, let conventional software execute it and verify the result.

This is not about avoiding AI. It is about placing AI precisely where it has the highest marginal value.

Context Is a Budget, Not a Storage Facility

The availability of very large context windows has created a dangerous illusion: if a model can technically receive hundreds of thousands of tokens, perhaps we should send them.

Usually, we should not.

Large context windows are useful, but they do not eliminate the need for context engineering. Sending an entire knowledge base, source-code repository, email chain or transaction history with every request increases cost and latency. It can also make the model’s task harder by surrounding the relevant evidence with noise.

Good context engineering asks:

• What does the model need to know for this specific decision?

• What information can be retrieved only when required?

• What can be represented as structured data rather than prose?

• Which historical messages are still relevant?

• Which tool descriptions does the model actually need?

• Can an intermediate result be summarized, compressed or discarded?

Retrieval-augmented generation should not mean “retrieve as much as possible.” It should mean retrieving the smallest sufficient evidence set.

The goal is not maximum context.

The goal is minimum sufficient context.

Token Efficiency in Software Development

AI coding assistants can produce substantial productivity gains, but they can also consume enormous contexts unnecessarily.

For example, asking a coding agent to inspect an entire repository for a change affecting one service may repeatedly load thousands of irrelevant files. Long instruction documents, verbose tool outputs and unrestricted exploration can further expand each interaction.

Our Token Diet approach to AI-assisted development includes:

• Scoping the task before asking the model to modify code.

• Providing repository maps instead of full repositories.

• Loading files incrementally based on dependency and relevance.

• Using static analysis, search, tests and compilers before model reasoning.

• Keeping project instructions concise and hierarchical.

• Separating planning, implementation and validation.

• Asking models to return patches or structured changes rather than reproducing complete files.

• Using deterministic linters and test suites to validate model-generated code.

• Routing simple edits to smaller models and reserving advanced models for architecture, debugging and complex refactoring.

The model should not rediscover information that the development environment already knows.

A compiler should identify syntax errors. A search tool should locate references. A test runner should execute tests. A language model should reason about the results.

Token Efficiency in Automation and Agentic Processes

The current AI market is fascinated with agents. Almost every workflow is now described as agentic, even when a conventional workflow engine would be faster, safer and less expensive.

Agents are valuable when a process genuinely requires adaptive planning, dynamic tool selection or reasoning under uncertainty. They are unnecessary when the sequence of operations is already known.

A fixed process such as:

1. Receive a document.

2. Validate its type.

3. Extract known fields.

4. Check the fields against a database.

5. Route the document according to established rules.

does not automatically need five collaborating agents.

It may need one model call for ambiguous extraction and deterministic software for everything else.

Agentic systems also have a hidden token multiplier. Every planning cycle, tool description, intermediate result, reflection step and agent-to-agent message may be added to the context. A process that appears to involve one user request can generate dozens of model interactions.

We therefore examine every proposed agent and ask:

• Is the task genuinely non-deterministic?

• Does the agent need access to every available tool?

• Can the tool list be selected dynamically?

• Can planning be replaced by a predefined state machine?

• Is a reflection step improving measured quality?

• Can tool results be filtered or summarized before returning to the model?

• Can the process stop earlier when confidence is already sufficient?

• Can repeated requests be cached safely?

An agent should earn its place in the architecture.

Token Efficiency in Information Analysis

Document analysis is another area where brute-force AI usage is tempting.

A straightforward implementation may divide every document into chunks, send all chunks to a model, generate summaries and then ask another model to summarize the summaries. This can work, but it may also process large quantities of irrelevant text.

A more efficient pipeline can first use conventional techniques to:

• Remove duplicate content, navigation elements and boilerplate.

• Detect document structure.

• Identify headings, tables and relevant sections.

• Filter records using metadata and lexical search.

• Apply embeddings only to appropriate content.

• Retrieve evidence according to the question.

• Group overlapping passages.

• Calculate whether sufficient evidence already exists.

• Send only the final evidence set for model analysis.

For structured datasets, SQL, dataframe operations and statistical code should perform calculations. The model should interpret the results, explain anomalies or help formulate the next analytical question.

Language models are poor substitutes for exact arithmetic and excellent partners for interpretation.

The distinction matters.

Techniques That Reduce Token Consumption

Several practical methods are becoming central to efficient AI architecture.

1. Model routing

Not every request requires the most capable model. A routing layer can send simple extraction, classification and transformation tasks to smaller models while escalating complex cases.

Routing should be based on measured quality, not merely intuition. The organization should know which task categories can be handled reliably by each model.

2. Prompt and prefix caching

Repeated system instructions, examples, schemas and document prefixes do not always need to be processed from scratch.

OpenAI, Anthropic and Google provide forms of prompt or context caching. OpenAI, for example, automatically applies prompt caching to eligible requests that share an exact prefix. This makes prompt structure important: stable instructions and reusable content should generally appear before dynamic information.

Recent research into long-running agentic workloads found that carefully designed caching strategies can materially reduce both cost and time to first token. The research also warns that naïvely caching a constantly changing full context may be less effective than maintaining stable cacheable sections and placing dynamic tool results later.

3. Semantic caching

Exact caching works only when requests repeat exactly. Semantic caching attempts to identify requests that are sufficiently similar for an existing result to be reused.

This can be powerful for repetitive support questions, analytics requests and internal knowledge queries. It must, however, be applied conservatively. Two questions may look semantically similar while differing in a legally, financially or operationally significant detail.

Tools such as GPTCache provide an open-source foundation for experimenting with semantic response caching.

4. Dynamic context assembly

Instead of maintaining one massive prompt, construct the prompt at runtime according to the current task.

This may include selecting:

• Relevant instructions.

• Appropriate examples.

• Required tools.

• User-specific context.

• Retrieved documents.

• Output schemas.

• Validation rules.

Dynamic assembly reduces token usage and limits the opportunity for irrelevant instructions to interfere with model behavior.

5. Conversation compaction

Long-running conversations and agents should not carry their complete history indefinitely.

Older interactions can be transformed into structured state, decisions, unresolved issues, and concise summaries. Raw history should remain accessible for audit purposes but should not automatically remain inside every model call.

6. Structured outputs

Requesting a constrained JSON object or schema-compliant response can reduce verbose output and simplify downstream processing. It also avoids additional model calls whose only purpose is to interpret an earlier free-form answer.

7. Output limits and stopping conditions

Many systems optimize input prompts while ignoring output consumption.

Applications should define the required level of detail, set realistic output limits and stop iterative processes when quality thresholds are reached. Generating five alternatives when the workflow needs one is not intelligence—it is waste.

8. Batch and asynchronous processing

Tasks that do not require an immediate answer can often be processed in batches or through lower-cost service tiers. This does not always reduce the number of tokens, but it can significantly improve the economics of those tokens.

9. Smaller and specialized models

A narrow model that performs one task reliably may be more efficient than a general frontier model. Depending on the use case, this can include compact hosted models, open-weight models, embedding models, classifiers or locally deployed models.

The correct comparison is not “Which model is smartest?”

It is:

Which architecture meets the required quality, latency, privacy and cost targets?

Measurement Comes Before Optimization

Token efficiency cannot be managed through provider invoices alone.

An invoice tells the organization how much it spent. It rarely explains which feature, customer, workflow, prompt version or agent step created that cost.

Each model interaction should therefore be traceable to its operational purpose.

At minimum, we recommend measuring:

• Input, output, reasoning and cached tokens where available.

• Cost per request and per completed business process.

• Model and provider.

• Application, customer, team and user.

• Prompt and workflow version.

• Number of model calls per process.

• Cache-hit rate.

• Retrieval volume and final context size.

• Tool-call count.

• Latency.

• Success and failure rates.

• Human correction or escalation rate.

• Quality metrics relevant to the task.

The most meaningful metric is rarely total token consumption.

A better metric is cost per successful business outcome.

A process that consumes more tokens but resolves a complex case correctly may be far more efficient than a cheaper process that creates rework, human intervention or customer dissatisfaction.

Token reduction must never become an objective disconnected from quality.

Open-Source Tools for Token Observability

A growing open-source ecosystem can help organizations measure and control model consumption.

Langfuse

Langfuse provides tracing, prompt management, evaluations and token-and-cost tracking for LLM applications. It can associate usage with traces, sessions, users and model calls, making it useful for understanding the complete economics of a workflow rather than individual API requests.

LiteLLM

LiteLLM can act as a common gateway across many model providers. Its spend-tracking capabilities record prompt tokens, completion tokens and cost and can attribute usage to keys, users, teams and custom request tags. This is particularly useful for organizations operating a multi-model architecture.

OpenLLMetry

OpenLLMetry provides open-source observability for LLM applications using OpenTelemetry conventions. It allows AI traces to be integrated with broader application and infrastructure observability rather than managed as an isolated AI monitoring layer.

Arize Phoenix

Phoenix is an open-source observability and evaluation platform for AI applications. It can be used to inspect traces, retrieval behavior, model calls and evaluation results, which helps teams identify expensive or low-quality stages in RAG and agentic pipelines.

OpenAI tiktoken

For systems using OpenAI-compatible tokenization, tiktoken provides a fast open-source tokenizer that can estimate prompt sizes before requests are submitted. Preflight token counting is useful for enforcing context budgets and preventing oversized requests.

GPTCache

GPTCache is an open-source semantic caching project designed to reuse answers to similar LLM requests. It can reduce repeated inference, although every cache implementation requires carefully designed similarity thresholds, invalidation rules and safeguards.

These products solve different parts of the problem. Token counting, tracing, cost attribution, evaluation, gateway management and caching are separate capabilities. In many enterprise environments, the correct solution will combine several of them.

The Token Diet Engineering Checklist

Before introducing or expanding a model call, we ask:

Necessity

• Is a language model actually required?

• Can a rule, query, search index, parser or traditional classifier solve the task?

Context

• What is the minimum information required?

• Are we sending duplicate, stale or irrelevant content?

• Can retrieval be more selective?

Model

• What is the smallest model that consistently meets the quality requirement?

• Is escalation to a stronger model possible only for difficult cases?

Workflow

• Is every agent and reasoning loop necessary?

• Can deterministic orchestration replace open-ended planning?

• Are tool results filtered before they return to the model?

Reuse

• Can prompts, intermediate results, retrieval results or final answers be cached?

• Is stable prompt content arranged to maximize provider-side caching?

Output

• Is the requested response longer than the business process requires?

• Can structured output eliminate another interpretation step?

Measurement

• Can we attribute the tokens to a customer, task and business result?

• Do we know whether the latest prompt or workflow version improved efficiency?

Quality

• Did token reduction preserve accuracy, safety and user experience?

• Are we optimizing cost per successful outcome rather than cost per API call?

Efficiency Is Part of AI Quality

For the last several years, the industry has focused primarily on what AI systems can do.

The next phase will focus on how intelligently they do it.

Enterprise AI cannot be judged solely by impressive demonstrations or benchmark scores. It must also be judged by operational reliability, explainability, latency, maintainability and unit economics.

A mature AI system does not use the largest model at every opportunity. It combines deterministic systems, retrieval, specialized models, powerful models and human oversight in the correct proportions.

That is the essence of the Token Diet.

Use conventional software wherever it is sufficient.

Use small models where the task is constrained.

Use powerful models where intelligence is genuinely critical.

Measure everything.

And never confuse consuming more intelligence with engineering a more intelligent system.