Provider economics: why reserved capacity, not price per token, drives cost and fit

A request with 1,025 tokens of input was rejected for asking for 65,025. The missing 64,000 were output tokens I never planned to generate. Groq's rate limiter counted them anyway, because it checks the headroom you reserve, not the tokens you use.

This essay is a deep-dive into one phase of my travel RAG case study: adding a second LLM provider behind the same interface as Claude. On paper it was a one-day job. In practice it surfaced four real bugs and one idea I now apply to every model decision: for RAG workloads, price per token is not the unit that decides whether a provider works. Reserved capacity is.

A note on dates: the measurements below were taken in June 2026, on Claude Sonnet 4.6 and Llama 3.3 70B. Since October 2026 the app runs Claude Sonnet 5.5 and Haiku 5.5, with GPT-OSS-120B on Groq after Groq retired Llama 3.3.

1.Planning

Why a second provider at all

By Phase 13 the app ran entirely on Claude: Sonnet for chat and trip planning, Haiku for small utility calls. A second provider was meant to prove three things:

• No lock-in. If a vendor changes pricing, terms or availability, the product survives.
• The abstraction is real. A Protocol only earns its keep when a second implementation plugs in without touching the routes.
• Honest comparison. The eval harness should score providers side by side on the same questions.

The plan vs what shipped

The architecture doc specified three providers. Two shipped.

The design

Everything hangs on one Python Protocol, LLMProvider. Routes never talk to a provider directly. They call the LLMOrchestrator, which picks a provider and a task budget. Each provider also declares a capabilities object, a small dataclass listing what it can and cannot do. The orchestrator reads that instead of hard-coding vendor knowledge.

The dashed box matters as much as the solid ones. The Protocol was designed so a local model can be added later without changing anything above it.

2.Implementation

A generic OpenAI-compatible client

I didn't use a vendor SDK for the second provider. OpenAICompatProvider is a plain httpx client for the OpenAI Chat Completions format, with base_url, api_key and model set per instance. Configured for Groq it runs Llama-3.3-70B; pointed elsewhere it works against DeepSeek, OpenRouter or Together AI with no code change.

Capabilities are declared, not assumed

Providers differ in what they can do, so each one declares it:

@dataclass(frozen=True, slots=True)
class LLMCapabilities:
    streaming: bool
    structured_output: bool
    tool_use: bool
    mcp: bool
    prompt_caching: bool
    max_output_tokens: int

Before any stream opens, the orchestrator calls assert_capability(...). A request that needs something the provider lacks fails with a clear HTTP 400, never a 500 and never an error halfway through a streamed answer.

One request, six steps

The routes stay identical for both providers. Everything that differs lives in data the provider declares or the orchestrator looks up.

In the UI

A dropdown in the header lists only the providers actually configured. Groq disappears when GROQ_API_KEY is unset. Every assistant message carries a coloured badge for its provider's quality tier. Switching provider starts a new conversation, with a banner that says so (TD-045), because a transcript re-sent to a different model is re-tokenised from scratch and may not fit.

Comparing providers honestly

The eval harness gained --provider and --all-providers, which print a side-by-side table. Because --all-providers multiplies token spend by the number of providers, running it without --limit requires an explicit --confirm-full-sweep flag. Langfuse prices both providers, so every trace shows its own cost: one Groq chat trace came to $0.000909 for 1,143 input and 297 output tokens.

3.Challenges

The smoke test after implementation turned up four real bugs and one environmental detour in a single session.

Bug 1: two bugs hiding each other

The dropdown was imported, rendered and backed by a working endpoint: curl /providers returned both providers. In the browser it showed nothing, and the console had no errors.

The dev server's proxy listed every earlier route but not the new /providers. The fetch fell through to Vite, which answered with its HTML page and status 200. Parsing that as JSON threw. The component then hit its defensive if (providers.length === 0) return null and rendered nothing at all. Either bug alone would have been visible. Together they were invisible to TypeScript, the build and the console.

The fix touched both layers. The selector now always renders one of four states: an error chip, a loading placeholder, a single-provider badge or a live dropdown. The incident also gave the frontend its first test suite, including a test that fails if the selector ever renders nothing.

Bug 2: one model's ceiling, applied to everyone

The orchestrator defaulted to max_tokens=64000, Claude's ceiling, for every provider. Groq's limit is 32,768. A clamp inside the orchestrator would have worked for Groq only. Instead, each provider now declares max_output_tokens in its capabilities, and the orchestrator sends min(requested, ceiling). Future providers need no orchestrator change.

Bug 3: the rate limit that wasn't about my prompt

With the clamp in place, Groq's free tier rejected the request: limit 12,000 tokens per minute, requested 35,846.

My first theory was that RAG prompts are big, so I added retrieval profiles. Groq got a compact profile of 3 paragraph-size chunks instead of 5 context-size ones. That was a sound design, and it saved about 2,000 tokens. The request still failed at 33,800.

This was the moment to stop guessing. I added one temporary log line at the chat route that broke the request into parts. The prompt was tiny: 425 tokens of system prompt, 595 of retrieved chunks and 5 from the user. Everything else was the output budget.

Groq's pre-check counts the prompt plus the full max_tokens you reserve. The size of the actual answer is irrelevant, because the check runs before anything is generated. A 200-token reply about Granada was blocked because I had reserved room for 32,768.

The fix was a per-task budget in the orchestrator: 4,000 tokens for chat, 8,000 for trip planning, which produces 2,000–4,000 tokens of JSON, and 512 for utilities like the eval judge. The same request dropped to 5,025 tokens and streamed cleanly. The retrieval shrink saved about 5% of the overrun; the budget fix saved about 85%.

4.Final results

Two providers now serve live chat traffic behind one interface, and the user picks between them in the UI. Neither route changed to make that happen.

One smoke test from the observability phase shows how different the two feel, with a caveat: Claude ran the rich retrieval profile and Groq the compact one, so this is not a controlled benchmark.

Groq is far faster and cheaper per call. Claude gives richer answers with caching, tools and trip planning. That trade-off is exactly why a user-facing choice, plus honest evals, beats picking one provider forever.

Deferred on purpose

• Ollama for local models (TD-044)
• Trip planning on Groq, which needs its own prompt and a reliability check of JSON mode on a 70B model (TD-049)
• Tool use on Groq until its tool-call behaviour is consistent enough to map onto Claude's (TD-048)
• Conversation carry-over when switching provider (TD-045)
• Tuning the budgets from real traffic. The 4,000-token chat budget is a reasoned guess. The next step is tracking p95 input and output tokens per provider and task, and setting budgets from that (TD-052).

Note: Groq decommissioned Llama 3.3 70B Versatile on 16 August 2026, and the Groq option stopped working. It is a live example of a challenge from The art of AI engineering: models have a short shelf life, and reliable serving means planning for the model underneath you to disappear. The Protocol made the replacement a configuration change: in October 2026 I swapped to GPT-OSS-120B on Groq. Someone still had to notice, pick a successor and re-run the evals.

5.Learnings

1. Cost per token is the wrong unit for RAG

Pricing pages quote cost per input and output token. That decides your bill. It does not decide whether the provider can serve your workload. Three things do:

• RAG prompts are not chatbot prompts. A chatbot message is a few hundred tokens. A RAG prompt carries a system prompt plus retrieved chunks, often 5,000 to 30,000 tokens.
• Rate limits are sized for chatbots. A free tier that feels generous for short messages can fit only one or two RAG requests a minute.
• Providers count reserved capacity differently. What you reserve can matter more than what you send.

With a 32,768-token reservation, nothing fits, whatever the prompt size. Cut the reservation to 512 and a compact RAG prompt fits almost 8 times a minute. The prompt size matters too, but only once the budget is sane.

2. Read how each provider counts, not just what it charges

The same max_tokens value is free on one provider and expensive on another.

Claude's behaviour is from Anthropic's rate-limit documentation. Groq's is what my own logs showed: the 413 error reported the prompt plus the reservation as the requested amount. Run your real workload against a provider's real limits before you commit. The pricing page is the start of the conversation, not the end.

3. Put provider differences in data, not in code

Both model-specific bugs came from the orchestrator assuming one vendor's numbers. Moving the ceiling into max_output_tokens and the budget into a per-task table turned each from a bug into a configuration value. The next provider declares its limits and inherits the clamp.

4. The Protocol earns its keep when something unexpected happens

In two days the same design absorbed two surprises: Amadeus closing sign-ups for hotel search, and a second LLM provider. Both became one-file changes. A Protocol named in an architecture doc proves nothing; a Protocol that survives a real event does.

5. Measure before you decide what to fix

The retrieval shrink was a good idea that fixed 5% of the problem. One log line, deleted afterwards, found the other 95% in a minute. Three times in this phase, a 30-second check (a log line, a grep, re-running a query on the other provider) ruled out the obvious theory before I spent hours on it.

6. Never render nothing

A component that returns null when it is meant to be visible hides every bug beneath it. Loading shows a placeholder, errors show an error, empty shows an empty state. Always something.

The short version: capacity is committed, not consumed. Before choosing a model for a RAG system, look at what you reserve, not only what you pay.