The Hidden Cost of Your AI Skills: Why We Need a 'Big O' Notation for Context

Recently, while reviewing the architecture of some AI agents, I stumbled upon something that kept me up at night: scripts embedded directly inside “skills” or tools.

I’m sure you’ve seen it. You define a skill so your LLM can check the weather, and you end up writing 100 lines of Python code that handle the HTTP request, error handling, and JSON parsing right into the same file or prompt that the model reads.

My first reaction, coming from a traditional software engineering background, was: This is highly coupled, untestable, and completely lacks reusability. But as I dug deeper, I realized there was a much more severe, LLM-specific problem: we are burning tokens and destroying the context window due to poor architecture.

Out of this frustration, a question was born: Why don’t we have something like Big O Notation to measure this?

The Problem: When the LLM Reads Your “Dead” Code

Every time you send your tool definitions to an LLM (via the System Prompt or Tool Schema), the model has to read all of that text to know what tools are available.

If you have embedded scripts or excessively long descriptions, you are paying for tokens that provide no executable value to the model itself. Even worse, LLMs suffer from the “Lost in the Middle” effect—they tend to forget information buried in the middle of long prompts. If there is too much code noise, the AI gets confused and fails to select the right tool.

The LLM doesn’t need to know how you cook the burger; it only needs to know how to order it.

The Solution: The “Wrapper” Pattern (Schema-Only)

The correct way to structure this is to think of skills as Adapters. The LLM should never see the implementation script. All it needs is the “contract” (the Schema).

What the LLM needs to see (Minimal Tokens):

{
  "name": "get_weather",
  "description": "Gets the current weather for a city.",
  "parameters": {
    "type": "object",
    "properties": {
      "city": { 
        "type": "string", 
        "description": "Name of the city" 
      }
    },
    "required": ["city"]
  }
}

The script (containing hundreds of lines of business logic, DB queries, third-party API calls) stays secure on your backend. The LLM only sees this small JSON schema, builds the arguments, and your backend executes the actual script natively.

Result: Zero implementation tokens. Zero output schema tokens. A clean, focused context.

Proposal: “Big C” Notation (Context Complexity)

In traditional software engineering, we use Big O Notation to measure the time and space complexity of an algorithm (O(1), O(n), O(n²)). In the world of LLMs, “space” is no longer RAM; it’s the context window. And “time” is the cost and latency of token processing.

Why haven’t we standardized a Big C Notation (Context Complexity) for AI systems?

Let’s define our variables:

• N: Number of available skills/tools in the system.
• S: Token size of the embedded script (business logic).
• s: Token size of the schema (name, description, inputs only).
• k: Number of dynamically loaded skills (Just-in-Time).

Using this lens, we can classify AI agent architectures into three complexity tiers:

1. The Anti-Pattern: O(N × S)

You embed scripts directly inside the skills and send all tools to the LLM on every turn. Your token consumption grows linearly but heavily. If you have 10 tools and each script takes up 500 tokens, you spend 5,000 tokens per turn just reading dead code. The system becomes slow, expensive, and unmanageable very quickly.

2. The Wrapper Pattern: O(N × s)

You separate the script from the tool definition. The LLM only reads the contract (the schema). Since s is significantly smaller than S, the baseline cost drops drastically. However, if N grows to 100 or 500 tools (e.g., an agent controlling an entire ERP), the complexity remains problematic because the model still has to read hundreds of schemas, leading to context pollution and selection errors.

3. The Optimal Pattern: O(k × s)

You use a secondary model or a semantic retriever (RAG) to determine which tools the user needs at that exact moment (Dynamic Tool Retrieval). Out of 500 tools, the router selects only the k necessary tools (where k is a small constant, e.g., k = 3). The main LLM always processes only the bare minimum. It doesn’t matter if you have 10 or 1,000 scripts on your backend; the context complexity remains constant relative to the task.

Visualizing the Architectures

loading…

Conclusion: From “Prompt Engineering” to “Context Engineering”

After discussing this with peers and researching, I realized the industry knows this is a problem (papers like Stanford’s Lost in the Middle prove it), but a formal notation to tackle it doesn’t exist yet. We are still measuring agents based on “vibes” rather than architectural metrics.

It’s time for developers to stop treating LLMs like a monolithic notebook where everything fits. “Skills” for the LLM should be menus only. The kitchen (the script) must be completely hidden behind your backend’s wall.

If your AI agent in production is failing, consuming too many tokens, or becoming unmanageable, review your architecture. Apply the Wrapper pattern, aim for an O(k × s) complexity, and start doing real Context Engineering.

---

What do you think? Will we ever adopt a standardized mathematical notation to measure the efficiency of our prompts and agents? Let me know in the comments below.