Auto Router v2 collapses complexity, semantic, and adaptive routing into a single auto_router/complexity_router. One config now covers heuristic scoring, LLM classification, lexical or semantic keyword rules, and Thompson-sampled tier pools.

The push came from the community. On discussion #32168, users pointed out that all three routing strategies should converge into a single Auto Router. One router with configurable signals and weights keeps the API simple while letting the routing engine evolve internally, instead of forcing you to pick a mode up front.

The operational half came from discussion #32172: predictable beats clever for debuggability. A fixed, versioned mapping from capability class to model is what makes "why did this response cost 4x today" answerable after the fact.

What v2 adds​

One config, all the knobs​

model_list:
  - model_name: smart-router
    litellm_params:
      model: auto_router/complexity_router
      drop_params: true
      complexity_router_config:
        tiers:
          SIMPLE:    ["gpt-4o-mini", "claude-haiku-4-5"]   # random-pick pool
          MEDIUM:    gpt-4o                                 # single pin
          COMPLEX:   claude-sonnet-5
          REASONING: gpt-5.5

        # optional: LLM classifier instead of heuristic scorer
        classifier_type: llm
        classifier_llm_config:
          model: claude-haiku-4-5-20251001
          timeout_ms: 2000

        # optional: keyword rules, escalate to highest matched tier
        keyword_tier_rules:
          - keywords: ["hi", "hello", "thanks"]
            tier: SIMPLE
          - keywords: ["kubernetes", "k8s", "istio"]
            tier: REASONING
        semantic_keyword_matching: true
        embedding_model: voyage-3-5
        match_threshold: 0.5

        # optional: append to the built-in technical keyword list
        custom_technical_keywords: [kafka, redis, postgresql, udp, dns]

        # optional: Thompson-sample within the tier's pool
        adaptive: true

        # optional: pin a session to its first-turn model (preserves prompt cache)
        session_affinity: true
        session_affinity_ttl_seconds: 3600

      complexity_router_default_model: claude-sonnet-5

Notes on the new pieces​

LLM classifier goes through the same Router instance, so credentials, budgets, and fallbacks apply. Timeout, empty content, or schema mismatch falls back to the heuristic scorer.

Keyword rules run before the scorer. Multiple matches escalate to the highest tier (SIMPLE < MEDIUM < COMPLEX < REASONING), so rule order does not silently change behavior. Semantic matching uses MAX aggregation (was MEAN), so one strong keyword match is not diluted by other utterances on the tier.

Adaptive turns tier pools into learning pools. Cold requests sample only inside the classified tier instead of collapsing on the cheapest model. Feedback attributes back to the model that actually served the previous turn, even when stateless routing picks a different one this turn.

Session affinity (opt-in) pins the first-turn model for a session and skips reclassification on later turns, so provider-side prompt caches keyed to that model do not get invalidated when a follow-up ("thanks!") would otherwise classify into a different tier (#33126). TTL defaults to 3600s. session_id comes from request metadata.

Decision log emits one greppable line per request:

ComplexityRouter: routing decision cause=complexity_scorer,      tier=SIMPLE,     score=-0.150, signals=['short (7 tokens)', 'simple (what is)'], routed_model=gpt-4o-mini
ComplexityRouter: routing decision cause=literal_keyword_match,  tier=REASONING,                                                                    routed_model=gpt-5.5
ComplexityRouter: routing decision cause=semantic_keyword_match, tier=REASONING,                                                                    routed_model=gpt-5.5
ComplexityRouter: routing decision cause=session_affinity_pin,                                                                                      routed_model=gpt-5.5

Fixes worth calling out​

drop_params, cache_control_injection_points, and any other litellm_params set on the auto router alias itself used to vanish when the router picked a tier. They now merge into the outbound request, without overriding anything the caller passed explicitly (#32974). Same PR fixes an Anthropic /v1/messages to Responses API tool_choice shape bug that broke Bedrock-backed complexity routers (reported in discussion #32168 by @icsy7867).

UI got a working Test Connection per tier (#32950) and required-tier inline validation (#32978).

Try it​

Existing complexity router configs keep working. To try v2, add keyword_tier_rules, classifier_type: llm, adaptive: true, session_affinity: true, or a list value on a tier to your existing complexity_router_config. Full reference on the Auto Routing docs page.

What's next​

Router plugins. From discussion #32168: a pipeline where each plugin receives the routing context, enriches it, and passes it on before Auto Router makes the final call. Plugins do not replace the router; they contribute structured signals (classification, policies, candidate filters, scores) that Auto Router combines.

Concrete end-to-end:

1. User sends a request.
2. Language plugin detects en.
3. Domain classifier labels it coding with 0.93 confidence.
4. Tenant policy limits allowed providers to OpenAI and Anthropic.
5. Budget plugin removes models exceeding the tenant's cost cap.
6. Auto Router picks the best remaining model from the enriched context.

Config sketch:

router_settings:
  plugins:
    - name: language-detector
    - name: domain-classifier
      params:
        provider: openai/gpt-5-mini
    - name: budget-policy
      params:
        daily_limit: 100
    - name: tenant-policy
    - name: custom-python
      path: ./plugins/my_router.py

The initial work landed in #32972; support for plugins on the proxy, and custom plugin files will be next.

Also on the list:

• Escalation ceilings on fallback chains. Per-request cap on escalations plus a cooldown once a key walks the chain N times, so a bad upstream cannot cascade into a bill.
• Attributable decisions. Stamp the routed model and routing-table version on every response, and export structured decision traces (candidates, scores, fallbacks, latency) through the standard logging integrations.

Running Auto Router in production and hitting these? Drop a note on discussion #32168.

Note: If you run Claude Code against Amazon Bedrock through LiteLLM on either v1.91.0 or v1.91.1, upgrade to v1.91.2 or higher.

Summary​

Between July 4 and July 10, proxies running v1.91.0 or v1.91.1 silently broke Anthropic prompt caching for Claude Code sessions routed through Amazon Bedrock's Invoke API. For the customers who reported it, warm-session cache hit rates dropped from roughly 90% to 25-45% and team daily spend rose 2-3x for the same usage. Requests kept returning 200s with correct completions; the only symptoms were the cache miss rate and the bill.

The cause: PR #31364 moved every role: "system" entry in messages into the top-level system field on the Invoke path, which invalidates every cache breakpoint past the tool definitions and system prompt. The fix shipped July 10 in v1.91.2 (#32578, #32831, #32882), with regression tests that fail on pre-fix code.

We own this outcome entirely. The trigger was a poorly documented change in how new Claude models and Claude Code use system messages, but customers run a gateway precisely so they do not have to track provider quirks. Translating requests faithfully, including their caching semantics, is our core job and here we fell short. This post explains exactly what happened, why our testing and review failed to catch it, and what we have changed so this class of regression does not ship again.

Background​

Three facts set up the incident:

1. Claude prompt caching is prefix based:
   1. the pricing of tokens in increasing cost is:
      1. cache read (0.1x)
      2. normal write (1x)
      3. cache write (1.25x for 5m ttl, 2x for 1h ttl)
   2. when Claude Code makes a new request, the Bedrock provider checks if any previous request was a truncated prefix of the current request. If so, it reads from the cache only up to that point.
2. Mid-conversation system messages are new:
   1. On May 28, 2026, Claude Opus 4.8 shipped as the first model accepting role: "system" entries inside messages (docs). This was documented on the Claude API docs on the same day.
   2. Claude Code (v2.1.154) began emitting them on May 28, 2026, with no mention in its changelog.
   3. Bedrock documented the same support by June 9, 2026 at the latest (the first archive.org capture; the page may have appeared as early as May 28).
3. Bedrock has two Anthropic APIs with different rules:
   1. Converse requires all system content in a top-level field; LiteLLM has hoisted it there since December 2024 (#7037).
   2. Invoke takes the native Anthropic Messages format, where models older than Opus 4.8 reject mid-conversation system entries with a 400 and newer models accept them.

What went wrong​

1. After May 28, Claude Code sessions on Bedrock Invoke began failing with 400s mid-session when two things were true:
   1. the model was older than Opus 4.8
   2. the model is served under an alias
      1. Claude Code detects capabilities by looking for version substrings like opus-4-7 or sonnet-4-6 in the model name.
      2. For example, an alias like bedrock-claude contains none, so Claude Code assumes the newest feature set and always emits mid-conversation system messages.
2. An enterprise customer worked around the 400s with a local patch that hoisted every system entry from messages into top-level system, mirroring the Converse behavior, and asked us to upstream it. We shipped it as #31364 in v1.91.0 on July 4.
3. The hoist fixed the 400s but by always hoisting the system entries in messages, that would invalidate the entire messages cache whenever a new mid-conversation system message was written by Claude Code, which we measured to happen every ~3 turns on average.
4. This greatly decreased the cache hit rate and increased the cache write rate, increasing spend.

Detection and response​

On July 8, an affected customer gave a detailed bug report of this regression. We reproduced it ourselves the same day end-to-end.

Three PRs fixed it, all released July 10 in v1.91.2 after extensive end-to-end testing, with regression tests that fail on pre-fix code:

1. #32578 disables mid-conversation system message hoisting on Invoke.
2. #32831 re-enables hoisting on models below Opus 4.8.
3. #32882 disables hoisting on Sonnet 5 and Fable 5, too.

Why our process did not catch this​

1. Testing the original patch was not end-to-end. We validated it with single-turn curl requests showing a 400 become a 200 that we assumed was the shape that Claude Code would use. That was not the case. We did not trace the root cause (we did not yet know mid-conversation system messages existed) or its caching implications, and treated it as a harmless edge case fix.
2. Review lacked the context to object. The human reviewer saw a small compatibility patch with passing tests and no explanation of why Claude Code started sending these kinds of mid-conversation system messages, and our AI review bots did not flag the caching implication either. Nobody in the loop had the info to connect the hoist to cache invalidation.
3. Cost regressions are silent. Every response was a 200 with a correct completion. The only signal was cache-read token counts, which nothing in our CI or monitoring measured.
4. The documentation was incomplete. The feature never appeared in the Claude Code changelog, and as of July 13 the Claude API docs still describe it as Opus 4.8 only (without mentioning Sonnet or Fable 5) and unavailable on Bedrock, which contradicts our empirical measurements.

What we are changing​

• Our e2e suite will gain a scripted multi-turn Claude Code session growing to roughly 250k tokens of context against real Bedrock, asserting cache reads grow monotonically and never collapse (started in #32963).
• We will set up a weekly automated load test that will flag anomalies in spend, cache reads and writes, turn latency, error rates, etc., so cost and performance regressions are detected and fixed before a release.
• Daily automated diffs of Anthropic's SDKs and docs alert us to new features that need translation support before customer traffic finds them.
• We dogfood LiteLLM internally and will set up monitoring for new request shapes, such as unknown anthropic-beta headers, and the same anomoly detction, which will alert us ahead of a release.
• Bug fixes now have a higher merge bar: validated means reproduced against the real client's traffic on their exact end-user application end-to-end and a complete understanding of the root cause; synthetic requests are not enough.

Known limitations​

1. Converse rejects system entries inside messages at any position, so on bedrock_converse we must still hoist, and Claude Code sessions routed through Converse still lose cached prefix on every mid-conversation system message. If you run Claude Code against Bedrock, route it through the Invoke path (bedrock/invoke/<model>). We are raising the API constraint with AWS.
2. We are testing whether the Vertex AI and Azure paths need equivalent hoisting and will update this post when we have more info.

To every team whose bill went up because of this: we are sorry. The value of a gateway is that this class of provider change gets absorbed by us instead of reaching you, and the tests, monitoring, and process improvements above are how we intend to keep it that way.

1. The MCP Gateway did not have a single class for credential resolution.
2. Pass-through APIs had high memory consumption.

Across the same window we shipped 134 bug fixes in total. This post covers the two big changes first, then the rest of the AI Eng and reliability work, the full breakdown, and what we are doing next.

MCP Gateway: a single credential resolver​

The MCP Gateway connects a user's AI app (Claude Desktop, Cursor, an agent) to the upstream MCP servers it wants to use, and it has to attach the right credential for each upstream.

Before, the MCP Gateway would infer the authentication method a user was trying to use based on the credentials they set. We have now built a credential resolver class that lets a user explicitly specify which auth method they are using. We believe this will significantly drive down a class of bugs users were reporting on MCPs.

Before: infer the auth method from whatever credentials are set​

Why this was bad: there was no single place that decided which credential to attach, and no error when the decision was ambiguous. Inferring from set fields meant two code paths could read the same server and disagree. The precedence order meant adding a field could silently change which credential won. And the silent fallback meant an unhandled case still sent something upstream instead of refusing. Ambiguity resolved to "attach a credential anyway" instead of "stop."

The types of bugs we saw from this:

• Tokens sent to the wrong upstream server.
• Duplicate or stale Authorization headers slipping through.
• MCP requests skipping the normal team, route, and key checks.
• Cached OAuth tokens going stale or crossing between users.
• Upstream URLs and secrets showing up in logs.

After: the user declares the auth method, and it fails closed​

Each mode has its own fully typed config, so there is no guessing from which fields are set and no precedence order. The match is exhaustive, so adding a mode without handling it fails the type checker, and an unhandled case raises instead of quietly attaching no auth.

AI Eng: LLM providers (27 fixes)​

AI Eng was focused on driving down reported bugs. Most of them were on new models (Claude 4.8, Opus 4.8, Bedrock Invoke). Here are the types of bugs we fixed:

Most of these were on Bedrock and the routing layer.

Performance: pass-through memory​

Pass-through APIs had high memory consumption. Large non-JSON pass-through downloads (batch-result files, binary and octet-stream downloads) were buffered whole in memory before being sent on. We changed this to stream the response chunk by chunk, so memory stays flat regardless of file size (#32386). This covers the provider pass-through routes (/vertex_ai/*, /bedrock/*, /openai/*, /anthropic/*, and others) and custom pass-through endpoints.

JSON responses still buffer by design, so spend logging and guardrails can inspect the body.

Two more fixes in the same spirit, don't pay for work nobody needs:

• Prometheus skips budget-metric DB lookups entirely when the gauges are no-ops (nothing is scraping them).
• The complexity router builds its semantic route index once under concurrent cold-start, instead of rebuilding it per request.

By the numbers​

All 134 fixes, by area:

These 134 are every merged fix: PR in the two weeks. One reported ticket often turns into several fix PRs, so this count is higher than the number of tickets in Linear.

Next goal: 95% end-to-end test coverage​

Most of these 134 bugs were caught late, in staging or from a user report. We want to catch them before they merge. We believe by investing in improving our e2e testing coverage we can significantly reduce the number of reported regressions from users on an upgrade.

We are learning from Meta's approach to fixing bugs fast and raising our testing bar.

Appendix​

PRs from this window.

MCP Gateway

Credential resolver:

• #32815 credential class merge (the single typed resolver)
• #32652 stale token invalidation
• #32715 semantic filter fail-closed

Pass-through and delegate credential modes:

• #31989 passthrough / delegate modes
• #32414 passthrough UI enum
• #32556 passthrough call relay
• #32752 configured client for passthrough
• #32735 no DCR persist for passthrough
• #32507 token exchange secret pairing

DCR bridge (LIT-4337):

• #32745 plumbing
• #32747 authorize relay
• #32753 facade
• #32804 UI
• #32527 DCR redirect_uri

Sealed envelope and delegate admission (LIT-4338):

• #32748 envelope module
• #32794 envelope edge consumer
• #32824 delegate admission

AI Eng (LLM providers)

Bedrock:

• #32882 flag Claude 4.8+ entries with supports_mid_conversation_system
• #32831 gate in-place system role messages for Claude Invoke
• #32578 keep mid-conversation system messages for Claude Invoke
• #32658 retain clear_tool_uses context-management edits and emit the beta header
• #32551 honor cache_control ttl on message-level cachePoint blocks
• #32538 preserve cache_control ttl on message-level cache points
• #32840 add jp.anthropic.claude-opus-4-8 to the model cost map
• #32389 resolve regional inference profiles to regional pricing

Anthropic:

• #32874 thread the real provider through capability probes (was pinned to anthropic)
• #32867 translate adaptive thinking/effort for pre-4.6 models
• #32833 strip @version suffix in model lookup

Routing and fallbacks:

• #32859 complexity router keyword tiers (max aggregation, blank-keyword hardening)
• #32943 complexity router logging and auth propagation, index built once
• #32873 fallback rules routing split (bare-Claude coverage, cost-map precedence, legacy schema)

Responses API:

• #32835 raise APIError on in-stream error events
• #32837 preserve reasoning_tokens through chat to responses
• #32034 idempotent response-id encoding (prevents MCP gateway double-encoding)

Cost, Vertex, Rerank:

• #32387 add gpt-realtime-2.1 models with regional uplift
• coerce string tiered-pricing costs and share the tier helper
• #32550 forward realtime health check params (Vertex)
• #32533 log rerank params at debug to stop leaking request content

Performance

• #32386 stream non-SSE pass-through responses instead of buffering in memory
• #32404 stop request params from clobbering merged target query params
• #32834 Prometheus skips budget-metric DB lookups when gauges are no-ops

Agenda​

• Product updates and roadmap progress
• Reliability and security updates
• Open Q&A with the team

Stability​

Our team is focused on stability: improving reliability and reducing regressions across releases. Follow the roadmap here: Stability roadmap.

How to contribute​

Add the topics and questions you'd like us to cover when you register below. We use your responses to set the agenda.

Register​

Register here: LiteLLM July Townhall Form

We will hold the townhall from 7:30 AM to 8:30 AM PT on Zoom.

For security, attendance is restricted to corporate emails. If you register with a non-corporate email, we will share the townhall slides and accompanying blog post after the event.

LiteLLM now supports the GPT-5.6 family. Route traffic to OpenAI's newest frontier models through the LiteLLM AI Gateway with no code changes.

GPT-5.6 introduces a new naming system where the number identifies the generation and the tier name identifies a durable capability level. gpt-5.6-sol is the flagship for complex reasoning and agentic workloads, gpt-5.6-terra is a balanced model for everyday work with performance competitive with GPT-5.5 at roughly half the cost, and gpt-5.6-luna is the fastest and most affordable tier. Per OpenAI, the family sets a new state of the art on agentic coding (Terminal-Bench 2.1) with broad gains in long-horizon biology and cybersecurity workflows. GPT-5.6 also adds a new max reasoning effort for the deepest single-agent thinking and an ultra mode that coordinates subagents on the most complex tasks.

Usage​

model_list:
  - model_name: gpt-5.6-sol
    litellm_params:
      model: openai/gpt-5.6-sol
      api_key: os.environ/OPENAI_API_KEY
  - model_name: gpt-5.6-terra
    litellm_params:
      model: openai/gpt-5.6-terra
      api_key: os.environ/OPENAI_API_KEY
  - model_name: gpt-5.6-luna
    litellm_params:
      model: openai/gpt-5.6-luna
      api_key: os.environ/OPENAI_API_KEY

docker run -d \
  -p 4000:4000 \
  -e OPENAI_API_KEY=$OPENAI_API_KEY \
  -v $(pwd)/config.yaml:/app/config.yaml \
  ghcr.io/berriai/litellm:v1.93.0-dev.2 \
  --config /app/config.yaml

curl -X POST "http://0.0.0.0:4000/chat/completions" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $LITELLM_KEY" \
  -d '{
    "model": "gpt-5.6-sol",
    "messages": [
      {"role": "user", "content": "Write a Python function to check if a number is prime."}
    ]
  }'

from litellm import completion

response = completion(
    model="openai/gpt-5.6-sol",
    messages=[
        {"role": "user", "content": "Write a Python function to check if a number is prime."}
    ],
)

print(response.choices[0].message.content)

# gpt-5.6-terra for balanced, cost-efficient everyday work
response = completion(
    model="openai/gpt-5.6-terra",
    messages=[
        {"role": "user", "content": "Summarize the key ideas in this design doc."}
    ],
)

print(response.choices[0].message.content)

# gpt-5.6-luna for the fastest, lowest-cost tier
response = completion(
    model="openai/gpt-5.6-luna",
    messages=[
        {"role": "user", "content": "Classify this ticket as bug, feature, or question."}
    ],
)

print(response.choices[0].message.content)

model_list:
  - model_name: gpt-5.6-sol
    litellm_params:
      model: azure/gpt-5.6-sol
      api_base: os.environ/AZURE_API_BASE
      api_key: os.environ/AZURE_API_KEY
      api_version: os.environ/AZURE_API_VERSION
  - model_name: gpt-5.6-terra
    litellm_params:
      model: azure/gpt-5.6-terra
      api_base: os.environ/AZURE_API_BASE
      api_key: os.environ/AZURE_API_KEY
      api_version: os.environ/AZURE_API_VERSION

from litellm import completion

response = completion(
    model="azure/gpt-5.6-sol",
    messages=[
        {"role": "user", "content": "Write a Python function to check if a number is prime."}
    ],
)

print(response.choices[0].message.content)

Responses API​

For agentic and multi-turn workflows, use /v1/responses to preserve reasoning state and output item metadata across turns.

curl -X POST "http://0.0.0.0:4000/v1/responses" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $LITELLM_KEY" \
  -d '{
    "model": "gpt-5.6-sol",
    "input": "Plan and write a Python script that scrapes a webpage and summarizes it."
  }'

Pricing​

Prices are per 1M tokens (USD), shown as short context (≤272K tokens) / long context (>272K tokens).

Global Azure OpenAI deployments (azure/gpt-5.6-*) match these OpenAI list prices. Regional deployments (azure/us/gpt-5.6-* and azure/eu/gpt-5.6-*) carry the standard 10% uplift on the base rate; LiteLLM tracks the difference automatically once you route through the regional model name.

Notes​

• For cost tracking on the GPT-5.6 models, hit the Reload Model Cost Map button in the Admin UI (or POST /reload/model_cost_map). Works on any LiteLLM version v1.76.0 or newer, with no container restart or image upgrade required.
• GPT-5.6 supports reasoning, function calling, parallel tool calls, vision (image input), prompt caching, web search, and structured output; see the OpenAI provider docs for advanced usage.
• The GPT-5.6 family launched in limited preview and OpenAI is expanding availability through the API and Codex; check your OpenAI account for model access.

Claude Code is one of the heaviest consumers of input tokens in a modern engineering org. Long tool loops, large file reads, and MCP catalogs with hundreds of tools push every request toward the top of the context window, and the bill scales with it.

If Claude Code already points at a LiteLLM proxy (via ANTHROPIC_BASE_URL), there are five levers the platform admin can pull to bring that cost down. None of them require a client-side change.

1. Budget windows + budget fallbacks​

Two knobs on the virtual key.

Budget windows cap how much the key can spend inside a rolling time period. Set max_budget (dollars) and budget_duration ("24h", "7d", "30d", etc). LiteLLM resets the counter automatically at the end of every window. You can stack windows too, e.g. $10/day AND $100/month, so one bad afternoon can't burn the whole month:

curl 'http://0.0.0.0:4000/key/generate' \
  --header 'Authorization: Bearer <your-master-key>' \
  --header 'Content-Type: application/json' \
  --data-raw '{
    "budget_limits": [
      {"budget_duration": "24h", "max_budget": 10},
      {"budget_duration": "30d", "max_budget": 100}
    ]
  }'

Budget fallbacks decide what happens once a per-model budget is exhausted. Instead of erroring at the developer's terminal, attach model_max_budget per model and a budget_fallbacks chain naming the cheaper models to reroute to. The request silently falls to the first fallback still under its own budget:

curl -X POST http://localhost:4000/key/generate \
  -H "Authorization: Bearer $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model_max_budget": {
      "claude-opus-4-8":   {"budget_limit": 20.0, "time_period": "1d"},
      "claude-sonnet-5":   {"budget_limit": 10.0, "time_period": "1d"},
      "claude-haiku-4-5":  {"budget_limit": 5.0,  "time_period": "1d"}
    },
    "budget_fallbacks": {
      "claude-opus-4-8":  ["claude-sonnet-5", "claude-haiku-4-5"],
      "claude-sonnet-5":  ["claude-haiku-4-5"]
    }
  }'

Once the developer burns $20 of Opus in a day, subsequent Opus requests silently reroute to Sonnet; if Sonnet is also tapped out, Haiku picks up. Fallback models without a model_max_budget entry are treated as unlimited.

→ Learn more: Budget Windows · Budget Fallbacks

2. Automatic Prompt Caching​

Claude's prompt cache reads a cache hit for roughly 10% of the price of a fresh input token, but only if the request marks the right message with cache_control. LiteLLM injects that marker for you: point cache_control_injection_points at the system message (or the second-to-last user turn), and every Claude Code call through the proxy carries the checkpoint without any client-side edit.

model_list:
  - model_name: claude-sonnet-4-5
    litellm_params:
      model: anthropic/claude-sonnet-4-5
      api_key: os.environ/ANTHROPIC_API_KEY
      cache_control_injection_points:
        - location: message
          role: system

router_settings:
  optional_pre_call_checks: ["prompt_caching"]

for automatically injecting this in all requests, do this

model_list:
  - model_name: claude-sonnet-4.5-20250929
    litellm_params:
      model: vertex_ai/claude-sonnet-4-5@20250929
      # ...

router_settings:
  default_litellm_params:
    cache_control_injection_points:
      - location: message
        role: system
  optional_pre_call_checks: ["prompt_caching"]

Turning on 'prompt_caching' as a pre call check, means if you run multiple deployments of the same Claude model, LiteLLM will intelligently route to the model deployment which was initially used for the request.

→ Learn more: Auto-Inject Prompt Caching Checkpoints · Claude Code - Prompt Cache Routing

3. Prompt Compression (Headroom)​

Prompt cache trims the static prefix; Headroom trims the dynamic middle. Tool outputs, file reads, database dumps, and RAG payloads get rewritten into a compressed form before they reach the model, and if the model actually needs the original bytes, a retrieve_headroom tool call fetches them on demand. Reported savings run 60-95% on the compressible portion of Claude Code traffic.

Headroom runs as a sidecar container next to LiteLLM. Register it as a pre_call guardrail and either flip default_on: true or attach it to per-developer virtual keys.

guardrails:
  - guardrail_name: headroom-compression
    litellm_params:
      guardrail: headroom
      mode: pre_call
      api_base: https://your-headroom-service
      default_on: true

The developer still exports ANTHROPIC_BASE_URL and runs claude; the only thing they notice is a smaller number on the spend log.

→ Learn more: Headroom guardrail setup guide

4. Defer MCP tools​

A Claude Code session that connects to five or six MCP servers can easily surface a few hundred tools, and every one of those tool schemas ships on every tools/list call. That is pure input-token overhead on a workload where the model uses two or three tools per turn.

Turn on mcp_tool_search_enabled on the virtual key, and LiteLLM replaces the full catalog with two virtual tools, mcp_tool_search and mcp_tool_call. The model searches by keyword, gets the ranked matches back, and calls the one it wants. The token cost of tool listing collapses from hundreds of schemas to two.

curl -X POST http://localhost:4000/key/generate \
  -H "Authorization: Bearer $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "object_permission": {
      "mcp_tool_search_enabled": true,
      "mcp_servers": ["github", "slack", "linear", "jira"]
    }
  }'

Ranking is token-overlap over name + description, so there is no embedding dependency to run. The access surface does not widen; search only returns tools the key was already allowed to call.

→ Learn more: MCP Tool Search

5. Auto routing​

Send every request to the smallest model that can handle it, so cheap requests never touch the expensive model. LiteLLM ships three flavors: Semantic (embedding match), Complexity (rule-based, zero external call), and Adaptive (learns from live traffic, beta).

Complexity router is the fastest to set up. Point Claude Code at smart-router and it classifies each request into a tier:

model_list:
  # Target models
  - model_name: gpt-4o-mini
    litellm_params:
      model: gpt-4o-mini

  - model_name: gpt-4o
    litellm_params:
      model: gpt-4o

  - model_name: claude-sonnet
    litellm_params:
      model: claude-sonnet-4-20250514

  - model_name: o1-preview
    litellm_params:
      model: o1-preview

  # Complexity router
  - model_name: smart-router
    litellm_params:
      model: auto_router/complexity_router
      complexity_router_config:
        tiers:
          SIMPLE: gpt-4o-mini
          MEDIUM: gpt-4o
          COMPLEX: claude-sonnet
          REASONING: o1-preview
      complexity_router_default_model: gpt-4o

→ Learn more: Complexity Router · Semantic Auto Routing · Adaptive Router

Stacking the levers​

The five features compose. Budget-based fallbacks bound the total spend regardless of what else you do. Prompt cache checkpoints and Headroom compression each shave a different slice of the request payload before it hits the model. MCP tool search cuts the tool schema overhead at the front of every turn. Auto routing sends every request to the smallest model that can handle it. Turn them on together and the same Claude Code workload runs on a fraction of the input tokens it did before, without touching a single developer machine.

Help us make this better​

We're actively investing in cost optimization across the whole stack. If you've got ideas, on auto routing, better cache heuristics, smarter budget policies, anything, join the discussion at litellm#32172.

LiteLLM now supports Claude Sonnet 5 on Day 0. Use it across Anthropic, Azure, Vertex AI, and Bedrock through the LiteLLM AI Gateway. Call it with the same OpenAI-compatible request you already use, and track spend, rate limits, and logging in one place.

What's new in Sonnet 5​

Sonnet 5 is the most agentic Sonnet model yet, with performance close to Opus 4.8 at a fraction of the price. A few things stand out for teams running it through a gateway:

• Opus-class quality, Sonnet pricing. Anthropic reports Sonnet 5 performs close to Opus 4.8 while costing far less, and is a substantial step up from Sonnet 4.6 on reasoning, tool use, coding, and knowledge work. (details from Anthropic)
• Built to run agents. It plans, drives tools like browsers and terminals, runs autonomously, and checks its own output without being asked, finishing complex tasks where earlier Sonnet models would stop short. Anthropic highlights gains on BrowseComp (agentic search) and OSWorld-Verified (computer use).
• Adaptive thinking only. Sonnet 5 decides how deeply to think on its own. You steer it per request with reasoning_effort or output_config.effort; fixed thinking budgets, temperature, top_p, and assistant message prefill are not supported by the model.
• $3 / MTok input and $15 / MTok output, with prompt caching at $0.30 / MTok (read) and $3.75 / MTok (write). Anthropic is running introductory pricing of $2 / MTok input and $10 / MTok output through August 31, 2026. On Bedrock, the us., eu., au., and jp. inference profiles carry the usual 10% regional premium while global. stays at base price; LiteLLM tracks every variant automatically.
• 1M-token context, up to 128K output tokens.
• One gateway, every surface. Vision, PDF input, computer use, tool calling, prompt caching, adaptive thinking, and structured output, all available across Anthropic, Azure, Vertex AI, and Bedrock with unified spend tracking, logging, and fallbacks.

Enabling Sonnet 5​

Sonnet 5 ships in the v1.92.0-dev.1 image (and every release after it). How you pick it up depends on where your proxy reads pricing from:

• Default (remote cost map): no upgrade needed. In the LiteLLM UI, open the Price Data tab under Models + Endpoints and click Reload Price Data (or, as a proxy admin, POST /reload/model_cost_map). This refetches the latest pricing from LiteLLM's cost map and re-registers provider routing in one step, so claude-sonnet-5 becomes available across Anthropic, Azure, Vertex AI, and Bedrock, even if you're on an older proxy version.

• Running LITELLM_LOCAL_MODEL_COST_MAP=true? The cost map is baked into the image, so the Reload button won't reach it. Pull v1.92.0-dev.1 or later to get the bundled Sonnet 5 metadata:

  docker pull ghcr.io/berriai/litellm:v1.92.0-dev.1
  

Usage​

Pick your provider below. Each tab wires up claude-sonnet-5 for that provider; the request you send afterward is identical everywhere.

model_list:
  - model_name: claude-sonnet-5
    litellm_params:
      model: anthropic/claude-sonnet-5
      api_key: os.environ/ANTHROPIC_API_KEY

docker run -d \
  -p 4000:4000 \
  -e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
  -v $(pwd)/config.yaml:/app/config.yaml \
  ghcr.io/berriai/litellm:v1.92.0-dev.1 \
  --config /app/config.yaml

model_list:
  - model_name: claude-sonnet-5
    litellm_params:
      model: azure_ai/claude-sonnet-5
      api_key: os.environ/AZURE_AI_API_KEY
      api_base: os.environ/AZURE_AI_API_BASE  # https://<resource>.services.ai.azure.com

docker run -d \
  -p 4000:4000 \
  -e AZURE_AI_API_KEY=$AZURE_AI_API_KEY \
  -e AZURE_AI_API_BASE=$AZURE_AI_API_BASE \
  -v $(pwd)/config.yaml:/app/config.yaml \
  ghcr.io/berriai/litellm:v1.92.0-dev.1 \
  --config /app/config.yaml

model_list:
  - model_name: claude-sonnet-5
    litellm_params:
      model: vertex_ai/claude-sonnet-5
      vertex_project: os.environ/VERTEX_PROJECT
      vertex_location: global

docker run -d \
  -p 4000:4000 \
  -e VERTEX_PROJECT=$VERTEX_PROJECT \
  -e GOOGLE_APPLICATION_CREDENTIALS=/app/credentials.json \
  -v $(pwd)/config.yaml:/app/config.yaml \
  -v $(pwd)/credentials.json:/app/credentials.json \
  ghcr.io/berriai/litellm:v1.92.0-dev.1 \
  --config /app/config.yaml

model_list:
  - model_name: claude-sonnet-5
    litellm_params:
      model: bedrock/anthropic.claude-sonnet-5
      aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
      aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
      aws_region_name: us-east-1

docker run -d \
  -p 4000:4000 \
  -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
  -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \
  -v $(pwd)/config.yaml:/app/config.yaml \
  ghcr.io/berriai/litellm:v1.92.0-dev.1 \
  --config /app/config.yaml

3. Test it!

The request is the same regardless of which provider you configured above:

curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $LITELLM_KEY' \
--data '{
  "model": "claude-sonnet-5",
  "messages": [
    {
      "role": "user",
      "content": "what llm are you"
    }
  ]
}'

Long-context agents burn most of their input budget on repeated tool output, retrieved chunks, and stale scratch state. Headroom intelligently rewrites that content so the model sees the same information at a fraction of the tokens.

If the model needs the full context, LiteLLM will also pass a 'retrieve_headroom' tool to the model, to retrieve the full context from Headroom.

How is it deployed?​

Headroom runs as a sidecar to LiteLLM. Client traffic still hits the LiteLLM gateway; LiteLLM invokes Headroom during the pre_call step, swaps in the compressed messages, and forwards the payload upstream. Clients and the LLM provider never talk to Headroom directly.

The benefit of this is two-fold

• Convenience: Users get 1 api base regardless of if they use prompt compression or not.
• Reliability: If Headroom goes down, your LLM calls are unaffected.

Compression works on both /v1/chat/completions and /v1/messages (Anthropic format), which makes the Claude Code rollout a one-liner for the admin: attach headroom-compression to a virtual key, hand it to the developer, and every request they make through ANTHROPIC_BASE_URL gets compressed automatically. No client-side change, no code diff.

Turn it on per key, per request, or globally via default_on: true. Confirm it ran by checking the x-litellm-applied-guardrails response header or the Guardrails panel in the Logs UI.

Get started: Headroom guardrail setup guide (requires LiteLLM v1.92.x or later; for testing ahead of the stable cut, grab the v1.92.0-dev.1 dev release)

Discussion: GitHub discussion #31816

Thank you to everyone who joined our June town hall.

Three numbers capture the month: 24 security fixes, 94 bug fixes, and 78 feature commits. The sections below break each one down, alongside our public commitment to zero reported regressions and the gradual migration of the LiteLLM gateway to Rust.

Security updates​

Last 4 weeks: by the numbers​

Bug bounty — now live​

We pay for security reports.

• Scope — the LiteLLM gateway and SDK.
• Submit via private vulnerability report on GitHub.
• Triaged by maintainers and the Veria Labs security team.

Automated review on every PR​

Every PR gets a security pass. Look for the Veria scan — it's a required check on every PR, built on Veria AI + zizmor + semgrep. False positives are flagged, never blocking.

What's next for security​

• Invest more in the bug bounty program.
• Improve code patterns during the stability sprint.

Stability updates​

The commitment: zero reported regressions by August 29th​

The goal:

• Close 20 reported bugs in core functionality.
• Fix root causes in 3 high-impact components.
• Ship a public progress report alongside the August 29 release.

94 bug fixes done​

Fixes shipped across five areas:

• Proxy core & resilience — 22 fixes
• UI + Auth / SSO — 22 fixes
• Cost, Budgets & Observability — 21 fixes
• MCP Gateway — 15 fixes
• Streaming / Realtime APIs — 14 fixes

What kinds of fixes shipped:

• Billing accuracy. Closed the gaps where spend slipped through — virtual-key limits are now enforced, and cached and tiered usage on Anthropic and Bedrock is priced correctly.
• Identity & access. Caller identity now resolves once into a single record, so team IDs and spend attribution stay correct and auth no longer fails open on DB errors.
• MCP reliability. Tools now list and call consistently across every auth method, with per-user credentials and proper OAuth token refresh.
• Resource leaks. Guardrails no longer re-initialize on every request, eliminating the runner leaks, latency spikes, and OOMs they caused.
• Resilience. Streaming requests recover cost on interruption, the proxy self-heals on dropped DB connections, and OTEL metrics no longer overload Splunk.

Root causes, not just symptoms:

• MCP authentication — 5 separate code paths, one per auth method, caused inconsistent tool listing and calling. Fix: a single unified code path resolves credentials across all auth methods.
• AI gateway auth — 5+ DB lookups per request to resolve key/user/team identity. Fix: caller identity resolves once into a single record — lookups cut roughly in half.
• UI forms — saving a form could overwrite unrelated fields. Fix: frontend and backend types are 100% in sync from a shared source, so only edited fields change on save.

Public timeline​

Bug triage is open and active on GitHub issue #30484.

• NOW — 20 bugs open in core. Triage active.
• JULY — MCP auth unified to a single code path. AI gateway identity lookups cut in half.
• AUGUST — UI form types synced end-to-end. No more silent field overwrites on save.
• AUG 29 — Public progress report ships with the release. Zero-regression target date.

Product updates​

78 feature commits in June​

Rust

• Rust workspace · Mistral OCR bridge
• OpenAI Realtime translation layer

Sandbox API

• E2B + OpenSandbox
• Unified code execution API

New models/providers

• TinyFish · Fal.ai · Fireworks AI
• Cloudflare Workers AI · MAI-Image-2.5

Performance: moving LiteLLM to Rust​

We're migrating the LiteLLM gateway to Rust, and the early numbers make the case:

Per-request overhead measured at 10 concurrent clients vs. a local mock upstream; throughput and memory under sustained load at 50 concurrent clients. Reproducible harness checked in.

How the migration works: a staged rollout, moving piece by piece from a pure Python SDK + FastAPI proxy, to Python driving Rust transforms via PyO3, to a FastAPI shell with pure Rust on the hot path, to an all-Rust async server (axum).

A gradual rollout — one route at a time, proven in production before the next begins. Same config, database, and API: nothing for you to change.

• Aug 15 — OCR routes: Mistral first, then all OCR.
• Sep 1 — /messages, then /chat/completions.
• Sep 15 — The router: load balancing, fallbacks, retries, cooldowns.
• Dec 1 — The full server: FastAPI thin shell, then pure Rust (axum).

Announcing our version policy​

Going forward, we'll maintain only the four most recent stable minor releases. This takes effect next Monday, June 29th. Our focus is ensuring stability on the most up-to-date product offerings — bookmark our Release Notes to stay current.

What's next​

Thank you again for all the questions and feedback. We'll keep sharing concrete progress updates as these efforts ship — especially as we approach the August 29 zero-regression milestone.

Hiring​

We are actively hiring across several roles — apply here if you're interested!

Thank you for using LiteLLM - Krrish & Ishaan

The OpenAI Responses and Chat Completions APIs let you declare a code_interpreter tool and the model runs Python inside an OpenAI-hosted container. That container is opaque, billed by OpenAI, and the code (often customer data) leaves your perimeter. LiteLLM now let's you intercept that tool call and runs it in a sandbox you control. The client request is unchanged.

Available starting LiteLLM v1.91.0.dev1. Check here for releases.

How the swap works​

Register a sandbox tool, enable the interceptor, then call the model the way you already do. When the model emits a code_interpreter tool call, LiteLLM creates a sandbox (E2B or OpenSandbox), executes the generated code, feeds the result back into the loop, and tears the sandbox down on completion. The response shape stays compatible with OpenAI's native code_interpreter_call.

Two backends are supported today: E2B for a managed sandbox, and OpenSandbox for self-hosted Docker-backed execution when the code or data cannot leave your network.

SDK​

import os, litellm
from litellm.sandbox.sandbox_tools import register_sandbox_tools
from litellm.integrations.code_interpreter_interception.handler import (
    CodeInterpreterInterceptionLogger,
)

os.environ["E2B_API_KEY"] = "e2b_..."
os.environ["OPENAI_API_KEY"] = "sk-..."

register_sandbox_tools([
    {
        "sandbox_tool_name": "my-e2b",
        "litellm_params": {
            "sandbox_provider": "e2b",
            "api_key": "os.environ/E2B_API_KEY",
        },
    }
])

litellm.callbacks = [
    CodeInterpreterInterceptionLogger(sandbox_tool_name="my-e2b")
]

response = await litellm.aresponses(
    model="openai/gpt-5",
    tools=[{"type": "code_interpreter", "container": {"type": "auto"}}],
    input="Product of first 6 primes. Just the number.",
)
print(response.output_text)

import os, litellm
from litellm.sandbox.sandbox_tools import register_sandbox_tools
from litellm.integrations.code_interpreter_interception.handler import (
    CodeInterpreterInterceptionLogger,
)

os.environ["E2B_API_KEY"] = "e2b_..."
os.environ["OPENAI_API_KEY"] = "sk-..."

register_sandbox_tools([
    {
        "sandbox_tool_name": "my-e2b",
        "litellm_params": {
            "sandbox_provider": "e2b",
            "api_key": "os.environ/E2B_API_KEY",
        },
    }
])

litellm.callbacks = [
    CodeInterpreterInterceptionLogger(sandbox_tool_name="my-e2b")
]

response = await litellm.acompletion(
    model="openai/gpt-4o-mini",
    messages=[{"role": "user", "content": "Product of first 6 primes. Just the number."}],
    tools=[{"type": "code_interpreter", "container": {"type": "auto"}}],
    max_agentic_loops=4,
)
print(response.choices[0].message.content)

The native code_interpreter tool is rewritten before it reaches OpenAI; on the chat path it becomes a litellm_code_execution function tool and LiteLLM appends each sandbox result as a role: tool message until the model returns a final answer.

Proxy​

Same swap behind the AI gateway, with no client-side change.

model_list:
  - model_name: gpt-5
    litellm_params:
      model: openai/gpt-5
      api_key: os.environ/OPENAI_API_KEY

sandbox_tools:
  - sandbox_tool_name: my-e2b
    litellm_params:
      sandbox_provider: e2b
      api_key: os.environ/E2B_API_KEY

litellm_settings:
  callbacks: ["code_interpreter_interception"]
  code_interpreter_interception_params:
    sandbox_tool_name: my-e2b

The OpenAI SDK keeps working unchanged. Point it at the proxy, declare code_interpreter, and the gateway handles the rest.

from openai import OpenAI

client = OpenAI(api_key="sk-1234", base_url="http://localhost:4000/v1")

response = client.responses.create(
    model="gpt-5",
    tools=[{"type": "code_interpreter", "container": {"type": "auto"}}],
    input="Product of first 6 primes. Just the number.",
)
print(response.output_text)

To run fully on-prem, swap the sandbox_tools entry to OpenSandbox:

sandbox_tools:
  - sandbox_tool_name: my-opensandbox
    litellm_params:
      sandbox_provider: opensandbox
      api_base: os.environ/OPEN_SANDBOX_API_BASE
      api_key: os.environ/OPEN_SANDBOX_API_KEY

OpenSandbox runs sandboxes locally with egress denied by default; flip allow_internet_access=True or pass an explicit network_policy when the code needs the network.

Why route it through your own sandbox​

You keep the OpenAI client contract while owning the execution layer. The generated code and any uploaded data stay inside the sandbox you operate, billing for execution stops going to OpenAI, and the same setup works for Responses and Chat Completions across any model the gateway routes to. Streaming, forced tool_choice, and concurrent requests are isolated per request and cleaned up on completion.

Full reference is in the sandbox docs.

Last Updated: June 2026

Over the past year, we have heard the same thing from our users and our community: they want the fastest, most lightweight AI gateway they can run. We have heard you. We are addressing it by moving LiteLLM to Rust, and committing to sub-1ms overhead with a sub-100MB memory binary you can deploy. By the end of this migration, you will get a pure Rust server that can serve 100% of your AI traffic, with every hot path operation, including auth and rate limiting, running in Rust.

The reason it matters: under real load, CPU and memory climb with concurrency, and pods get OOM-killed at the worst time. Today the LiteLLM Python proxy peaks around 359MB of memory under load, and that cost multiplies across every pod, region, and retry you run.

We are already seeing the payoff in benchmarks. The Rust gateway serves about 15x the throughput (453 to 6,782 requests per second) on about 11x less memory (359MB to 32MB), and cuts per-request overhead from about 7.5ms on the Python path to about 0.05ms, well under the 1ms we commit to.

What you get​

You deploy a single Rust binary. It uses about 65MB of memory, gateway overhead stays under 1ms, and nothing in your setup changes: same config.yaml, same database, same client API, same providers. You keep LiteLLM's coverage of 100+ LLM providers behind one OpenAI-compatible API, with /chat/completions, /messages, /responses, and every other LLM endpoint LiteLLM supports today, now as the fastest and most lightweight LLM gateway you can self-host.

This is not a v2 and not a rewrite. There is no new major version to migrate to and nothing for you to change. The runtime under the hot path gets faster and lighter while your config stays exactly where it is.

We ship this the careful way. Each route moves to Rust only after it passes our full parity and end-to-end test suite, and it runs in production before the next route starts. Stability is the priority, and we target zero regressions on every release.

How fast is the LiteLLM gateway? A throughput, overhead, and memory benchmark​

Per-request overhead. We built a small harness: a mock upstream, a thin Rust forwarding gateway (axum), the same forwarding path running through LiteLLM today (litellm.acompletion over uvicorn), and a load client that times each request in microseconds. At 10 concurrent clients against the same mock, the Rust gateway adds about 0.05ms of overhead per request; the LiteLLM Python path adds about 7.5ms. That is roughly 150x lower, and well under the 1ms we commit to.

Sustained load. Against the current LiteLLM Python proxy on the same /v1/responses workload at 50 concurrent clients, the Rust path served about 15x the throughput on about 11x less memory.

The overhead harness (mock, gateway, load client) is checked in next to this post under benchmark/, and the summarized numbers are in rust_proxy_benchmark_results.csv, so you can reproduce the sub-1ms result. This measures the gateway forwarding path (request transform, forwarding, response handling), not a full production workload.

What stays the same​

Nothing you depend on changes. The migration is invisible from the outside:

• Your Python SDK keeps the exact same interface; the same calls now run on Rust bindings underneath.
• Your config.yaml is unchanged.
• Your database and schema are unchanged.
• Your client API and request/response shapes are unchanged.
• Your providers, routing, and keys are unchanged.

You get lower memory and lower overhead, and you do nothing to get it.

How the migration works​

If you just want the outcome, you have it above. The rest of this post is for engineers who want to see how we move the gateway to Rust without breaking anything.

The core idea is a clean split. We build one Rust core that only transforms data: it turns your request into a provider request, turns the provider response back, transforms stream chunks, counts tokens, and normalizes errors. It never opens a socket, reads a secret, or writes to your database. The host process does all of that. That separation is what lets us put Rust into production without rewriting the server, because Python keeps doing the I/O while Rust takes over the translation.

One route at a time, proven in production​

We never flip a whole endpoint at once. For each route we prove one provider first, roll it out to every provider on that route, and only then start the next route. The smallest, lowest-risk route goes first.

In Stage 1 the server does not change shape. Python still serves traffic and does the I/O, but hands translation to the Rust core through a flag-gated binding, per provider. A parity check enforces identical output before any provider turns on, and if the flag is off the existing Python path runs unchanged.

The routes move in order of risk:

• OCR first. Start with Mistral OCR, the smallest surface: no streaming, tiny schema, few params. Once it matches the Python output byte for byte in production, roll out to all OCR providers, then move the route into the Rust core. Integration risk is retired here before any larger endpoint moves.
• /v1/messages next. This adds streaming: SSE parsing, chunk emission, usage accounting, token cost. One provider first, then all, then the route into Rust.
• /chat/completions after that. The largest surface, taken on only once streaming is proven: tools, function calling, multimodal, and the full optional-param matrix.
• Major providers. Azure, then Bedrock, then Vertex, by traffic volume. Auth-coupled providers get signed headers from the host (boto3 / google-auth first, native Rust later). Long-tail providers keep running on Python.

Onto a Rust server​

Once the routes run on Rust, the router moves too: routing, fallbacks, retries, and cooldowns, with state in Redis. Then the server itself moves in two steps.

• FastAPI as a thin shell. FastAPI still terminates HTTP and runs auth, rate-limit, and callbacks, but the entire forwarding path is a single call into Rust.
• Pure Rust server. A native server (axum / hyper) runs the forwarding path with no Python on the hot path. Your custom Python plugins (auth, guardrails, callbacks, SSO) keep working in an optional sidecar, so nothing breaks. We roll it out with shadow traffic and a percentage cutover.

The end state is a pure Rust data plane. Customer Python plugins keep running in the sidecar, so it is non-breaking. Removing Python entirely would require porting plugins to a Rust or WASM interface, which is a breaking change we are deferring.

Why this order​

• The OCR route retires integration risk on the smallest surface.
• /v1/messages retires streaming risk before the largest parameter set.
• /chat/completions is taken on only after streaming is proven.
• By the time the server moves, the core, providers, and router are already running in production through the SDK, so the server work is mostly plumbing.

Every step ships to real users before the next begins, with the parity check as the gate.

Timeline​

We move one function at a time, smallest first, and only after each step passes our test suite.

Frequently asked questions​

Is LiteLLM the fastest LLM gateway?​

That is the goal of this work. With the Rust hot path, LiteLLM targets sub-1ms gateway overhead and a sub-100MB binary, matching compiled-language gateways while keeping coverage of 100+ providers behind one OpenAI-compatible API. In our benchmark the Rust gateway adds about 0.05ms of overhead per request, versus about 7.5ms for the LiteLLM Python path today, and serves 6,782 requests per second under load at 31.7MB peak memory. Gateway overhead is usually a small fraction of total model latency, so it matters most for high-throughput, low-latency workloads like classification and embeddings at scale.

Is LiteLLM slow?​

Gateway latency and throughput depend on how you deploy the proxy: worker count, concurrency settings, and whether logging callbacks run on the hot path. Tuned, the Python proxy serves production traffic across hundreds of providers today. Moving the hot path to Rust pushes the floor lower still: in our reproducible benchmark the Rust gateway adds about 0.05ms of overhead per request, versus about 7.5ms for the LiteLLM Python path, and serves 6,782 requests per second at 31.7MB peak memory.

Is LiteLLM limited by the Python GIL?​

The GIL only matters for CPU-bound work on the request path, and the gateway is mostly I/O. LiteLLM scales today by running multiple workers. The Rust migration removes the question for the hot path: request transforms, streaming, and routing run in the Rust core and router, outside the GIL, with no first-party Python on the forwarding path in the end state.

How much memory does the LiteLLM gateway use?​

The Python proxy peaked at 358.9MB under our load test. The Rust end state targets roughly 65MB. Lower, bounded memory is the main reason for this work: it reduces the high-CPU and OOM failures that show up under concurrent load.

Are these benchmarks reproducible?​

Yes. The overhead harness (a mock upstream, a thin Rust gateway, and a load client that times each request in microseconds) is checked in under benchmark/, alongside the summarized CSV. Same upstream and payload for both runtimes; the only variable is Python versus Rust.

Will the Rust gateway be a breaking change?​

No. Config, database schema, and the client API contract stay the same. The runtime under the hot path changes gradually, route by route, behind passing parity and end-to-end tests.

We are hiring Rust engineers​

We are building this with a small team and looking for Rust engineers who want to work on the hot path of an AI gateway that serves 100+ providers. If that sounds like you, come build it with us.

References​

• How Datadog migrated their static analyzer from Java to Rust
• How GitGuardian migrated the heart of their platform to Rust
• LiteLLM AI Gateway, full feature overview
• Load balancing and routing across 100+ LLM providers

Why we're doing this​

Maintaining older lines means carrying every fix back to keep them all in parity. That overhead grows with the number of lines we keep alive, not the number of fixes we make. Our focus is ensuring the most up-to-date product offerings are stable and working for you. Because of this, LiteLLM is focusing on the four most recent stable minor lines going forward.

How the rolling window works​

This shift in focus takes effect Monday, June 29, 2026.

A minor line is a release series written as 1.89.x, covering every patch in it: 1.89.0, 1.89.1, 1.89.2, and any later ones. We support the four most recent lines and every patch inside each of them.

Today the four supported lines are 1.89.x, 1.88.x, 1.87.x, and 1.86.x. Everything 1.85.x and earlier has reached end of life and will no longer actively receive updates. The window rolls forward: when 1.90.x ships, 1.86.x rolls out and the supported set becomes 1.90.x, 1.89.x, 1.88.x, and 1.87.x. With a new line about every week, that works out to roughly a month of coverage per line.

What this means for you​

To stay supported, pin to a line and take its patches, then move up before it ages out. Patching within a line is a drop-in; moving up a line is where you'd check the release notes for changes. Enterprise customers who need longer coverage can reach out, and for rare high-severity issues we'll use our judgment and may patch outside the window.

How to stay current​

The best way to stay up to date on these changes is to bookmark our release notes. We update it as new versions ship, so you can see the latest stable line and the three behind it that are still supported.

Why this matters​

Semantic caching stores responses by the meaning of a prompt rather than an exact string match, so a reworded request can still hit the cache and skip a paid model call. Until now LiteLLM's semantic cache was built on RedisVL, which depends on RediSearch's FT.* vector API. RediSearch is not available on Redis OSS or on ElastiCache for Redis OSS, which left teams standing up Redis Stack or Qdrant just to get semantic caching. With Redis moving to a source-available license, more teams are standing up Valkey instead, and ElastiCache for Valkey is a common managed target.

Valkey ships vector search through the valkey-search module, and ElastiCache for Valkey exposes it. LiteLLM's new backend talks to valkey-search directly over the Redis protocol, so semantic caching on ElastiCache for Valkey works without RediSearch, Redis Stack, or Qdrant in the path.

How it works​

The valkey-semantic backend builds its own vector index from the field types valkey-search supports, a tag field that isolates each cache key's scope and an HNSW vector field for the prompt embedding, then runs a KNN query at lookup time and returns the cached response when the cosine similarity clears your threshold. Prompt extraction, embedding generation, and response handling are shared with the existing Redis semantic cache, so behavior matches the Redis path including per-request scope isolation. Connections resolve from VALKEY_HOST, VALKEY_PORT, and VALKEY_PASSWORD, falling back to the REDIS_* equivalents, and passwordless clusters are supported for IAM or no-auth setups.

Get started​

Add the cache to your config.yaml:

litellm_settings:
  cache: True
  cache_params:
    type: valkey-semantic
    host: os.environ/VALKEY_HOST
    port: os.environ/VALKEY_PORT
    valkey_semantic_cache_embedding_model: openai-embedding
    similarity_threshold: 0.8

For ElastiCache with encryption in transit, pass a rediss:// URL through cache_params.redis_url instead of host and port. To try valkey-search locally, the bundled image has the module ready:

docker run -d -p 6379:6379 valkey/valkey-bundle:8.1

See the caching docs for the full setup, including the SDK usage and the parameter reference.

We are hosting our June townhall on Thursday, 25 June at 7:30 AM PST.

Agenda​

• Product updates and roadmap progress
• Reliability and security updates
• Open Q&A with the team

Stability​

Our team is focused on stability: improving reliability and reducing regressions across releases. Follow the roadmap here: Stability roadmap.

How to contribute​

Add the topics and questions you'd like us to cover when you register below. We use your responses to set the agenda.

Register​

Register here: LiteLLM June Townhall Form

We will hold the townhall from 7:30 AM to 8:30 AM PST on Zoom.

For security, attendance is restricted to corporate emails. If you register with a non-corporate email, we will share the townhall slides and accompanying blog post after the event.

We're kicking off a stability sprint for LiteLLM with one bar in mind: 0 reported regressions by our next release on August 29th. The sprint has 2 goals:

• Close 20 reported bugs in core functionality - here
• Address the root cause of underlying bugs in 3 core components - MCP, Gateway, and UI

What class of bugs are we driving down?​

Over this sprint we're driving down 3 classes of bugs:

• MCP Authentication: View/List Tools did not consistently work across all our supported MCP auth methods.
• Gateway Authentication: Team IDs are not reliably on every request trace. As a result, some requests and budgets are not accurately tracked to a team.
• UI Forms: Today when users hit save on a form, it can accidentally wipe out other fields on the form, across keys, teams, and users.

MCP Authentication: Consistent behavior across all MCP Authentication Methods​

Solution: We've identified that the root cause of bugs across MCPs is that we maintain 5 different code paths, one per authentication method. To fix this and restore connection reliability, we're refactoring this into one code path that resolves MCP credentials across all supported authentication methods. The result: tools list and call reliably, no matter which auth method you use.

AI Gateway Authentication: Spend is always attributed to the right team​

Solution: We identified that the authentication layer makes 5+ DB lookups to resolve the exact key, user, team, and team member making a request. To fix this, we're resolving caller identity once, into a single record that every check and log reads from. This cuts identity lookups roughly in half, and means spend is always attributed to the team that made the request.

UI: Edits change only what you touched​

Solution: One of the root causes of UI bugs on form save is that our data shapes across the frontend and backend are not consistent. To fix this, we're refactoring so frontend and backend types are 100% in sync and read from the same source of truth. The result: a save changes only the field you edited, nothing else.

How you'll know it worked​

We'll report back at the August 29th release on exactly where each of these stands. You shouldn't have to take our word for it.

Why now​

We've grown fast. And fast growth in a complex system means bugs accumulate if you're not deliberate about paying them down. This sprint is us being deliberate.

We're also being public about it because you deserve to know what's being fixed and when. Stability is infrastructure. We're treating it that way.

Want us to fix something?​

Every item above came from real user reports. If there's a bug affecting you that isn't on this list, comment on the GitHub issue. We're actively triaging!

LiteLLM now supports Claude Fable 5 on Day 0. Use it across Anthropic, Azure, Vertex AI, and Bedrock through the LiteLLM AI Gateway. Call it with the same OpenAI-compatible request you already use, and track spend, rate limits, and logging in one place.

What's new in Fable 5​

Fable 5 is Anthropic's first publicly available Mythos-class model, priced at 2x Opus 4.8. A few things stand out for teams running it through a gateway:

• The frontier, now public. Anthropic reports Fable 5 is state-of-the-art on nearly all tested benchmarks, and the highest scorer among frontier models on Cognition's frontier coding benchmark, even at medium reasoning effort. (details from Anthropic)
• Built for long-running work. A 1M-token context window and up to 128K output tokens, with focus that holds across millions of tokens in long-horizon agentic tasks.
• Adaptive thinking only. Fable 5 decides how deeply to think on its own. You steer it per request with reasoning_effort or output_config.effort; fixed thinking budgets, temperature, top_p, and assistant message prefill are not supported by the model.
• $10 / MTok input and $50 / MTok output, with prompt caching at $1.00 / MTok (read) and $12.50 / MTok (write). On Bedrock, the us. and eu. inference profiles carry the usual 10% regional premium while global. stays at base price; LiteLLM tracks every variant automatically.
• A fallback you might notice. On flagged cybersecurity and biology requests (under 5% of sessions, per Anthropic), the response is served by Opus 4.8 instead.
• One gateway, every surface. Vision, PDF input, computer use, tool calling, prompt caching, adaptive thinking, and structured output, all available across Anthropic, Azure, Vertex AI, and Bedrock with unified spend tracking, logging, and fallbacks.

Before you flip it on: provider opt-ins​

Fable 5 requires a data sharing opt-in on some clouds; prompts are shared with Anthropic and retained for up to 30 days.

• Bedrock: set your account's data retention mode to provider_data_share, and invoke through an inference profile (us., eu., or global. prefix); direct model ID invocation is not supported.
• Vertex AI: enable Anthropic data sharing for your project and accept the Fable 5 terms in Model Garden.
• Azure AI Foundry: create a claude-fable-5 deployment; the model's TPM quota meter starts at 0 on some subscriptions, so you may need a quota request first.

Enabling Fable 5​

Fable 5 ships in the v1.89.0-rc.2 image (and every release after it). How you pick it up depends on where your proxy reads pricing from:

• Default (remote cost map): no upgrade needed. In the LiteLLM UI, open the Price Data tab under Models + Endpoints and click Reload Price Data (or, as a proxy admin, POST /reload/model_cost_map). This refetches the latest pricing from LiteLLM's cost map and re-registers provider routing in one step, so claude-fable-5 becomes available across Anthropic, Azure, Vertex AI, and Bedrock, even if you're on an older proxy version.

• Running LITELLM_LOCAL_MODEL_COST_MAP=true? The cost map is baked into the image, so the Reload button won't reach it. Pull v1.89.0-rc.2 or later to get the bundled Fable 5 metadata:

  docker pull ghcr.io/berriai/litellm:v1.89.0-rc.2
  

Usage - Anthropic​

model_list:
  - model_name: claude-fable-5
    litellm_params:
      model: anthropic/claude-fable-5
      api_key: os.environ/ANTHROPIC_API_KEY

docker run -d \
  -p 4000:4000 \
  -e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
  -v $(pwd)/config.yaml:/app/config.yaml \
  ghcr.io/berriai/litellm:v1.89.0-rc.2 \
  --config /app/config.yaml

curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $LITELLM_KEY' \
--data '{
  "model": "claude-fable-5",
  "messages": [
    {
      "role": "user",
      "content": "what llm are you"
    }
  ]
}'

Usage - Azure​

model_list:
  - model_name: claude-fable-5
    litellm_params:
      model: azure_ai/claude-fable-5
      api_key: os.environ/AZURE_AI_API_KEY
      api_base: os.environ/AZURE_AI_API_BASE  # https://<resource>.services.ai.azure.com

docker run -d \
  -p 4000:4000 \
  -e AZURE_AI_API_KEY=$AZURE_AI_API_KEY \
  -e AZURE_AI_API_BASE=$AZURE_AI_API_BASE \
  -v $(pwd)/config.yaml:/app/config.yaml \
  ghcr.io/berriai/litellm:v1.89.0-rc.2 \
  --config /app/config.yaml

curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $LITELLM_KEY' \
--data '{
  "model": "claude-fable-5",
  "messages": [
    {
      "role": "user",
      "content": "what llm are you"
    }
  ]
}'

Usage - Vertex AI​

model_list:
  - model_name: claude-fable-5
    litellm_params:
      model: vertex_ai/claude-fable-5
      vertex_project: os.environ/VERTEX_PROJECT
      vertex_location: global

docker run -d \
  -p 4000:4000 \
  -e VERTEX_PROJECT=$VERTEX_PROJECT \
  -e GOOGLE_APPLICATION_CREDENTIALS=/app/credentials.json \
  -v $(pwd)/config.yaml:/app/config.yaml \
  -v $(pwd)/credentials.json:/app/credentials.json \
  ghcr.io/berriai/litellm:v1.89.0-rc.2 \
  --config /app/config.yaml

curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $LITELLM_KEY' \
--data '{
  "model": "claude-fable-5",
  "messages": [
    {
      "role": "user",
      "content": "what llm are you"
    }
  ]
}'

Usage - Bedrock​

model_list:
  - model_name: claude-fable-5
    litellm_params:
      model: bedrock/converse/us.anthropic.claude-fable-5
      aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
      aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
      aws_region_name: us-east-1

docker run -d \
  -p 4000:4000 \
  -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
  -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \
  -v $(pwd)/config.yaml:/app/config.yaml \
  ghcr.io/berriai/litellm:v1.89.0-rc.2 \
  --config /app/config.yaml

curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $LITELLM_KEY' \
--data '{
  "model": "claude-fable-5",
  "messages": [
    {
      "role": "user",
      "content": "what llm are you"
    }
  ]
}'

Advanced Features​

Adaptive Thinking​

curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $LITELLM_KEY' \
--data '{
  "model": "claude-fable-5",
  "messages": [
    {
      "role": "user",
      "content": "Solve this complex problem: What is the optimal strategy for..."
    }
  ],
  "reasoning_effort": "high"
}'

curl --location 'http://0.0.0.0:4000/v1/messages' \
--header 'x-api-key: sk-12345' \
--header 'content-type: application/json' \
--data '{
    "model": "claude-fable-5",
    "max_tokens": 16000,
    "thinking": {
        "type": "adaptive"
    },
    "messages": [
        {
            "role": "user",
            "content": "Explain why the sum of two even numbers is always even."
        }
    ]
}'

Effort Levels​

Claude Fable 5 supports the full effort ladder: low, medium, high (default), xhigh, and max. These give you finer-grained control over how much reasoning the model applies to a task. Pass the effort level via the output_config parameter.

On Bedrock, output_config.effort caps at xhigh; the other providers accept the full ladder up to max.

curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $LITELLM_KEY' \
--data '{
  "model": "claude-fable-5",
  "messages": [
    {
      "role": "user",
      "content": "Explain quantum computing"
    }
  ],
  "output_config": {
    "effort": "max"
  }
}'

import openai

client = openai.OpenAI(
    api_key="your-litellm-key",
    base_url="http://0.0.0.0:4000"
)

response = client.chat.completions.create(
    model="claude-fable-5",
    messages=[{"role": "user", "content": "Explain quantum computing"}],
    extra_body={"output_config": {"effort": "max"}}
)

from litellm import completion

response = completion(
    model="anthropic/claude-fable-5",
    messages=[{"role": "user", "content": "Explain quantum computing"}],
    output_config={"effort": "max"},
)

curl --location 'http://0.0.0.0:4000/v1/messages' \
--header 'x-api-key: sk-12345' \
--header 'content-type: application/json' \
--data '{
    "model": "claude-fable-5",
    "max_tokens": 4096,
    "messages": [
        {
            "role": "user",
            "content": "Explain quantum computing"
        }
    ],
    "output_config": {
        "effort": "max"
    }
}'

Effort level guide:

Last updated: June 2026

Agent infrastructure is already separating into three layers: models, harnesses, and runtimes. We believe a fourth layer will emerge: the unified agent control plane. This will allow calling agents living in different agent runtimes, all from 1 place.

The reason is that companies will not run every agent on one runtime. Coding agents may run on Bedrock AgentCore or Claude Managed Agents. Data agents may run inside Elastic, Databricks, or Snowflake. Internal workflow agents may run on custom infrastructure. The control plane emerges because companies want one place where all of these agents can be used, regardless of where they were built or run.

But a registry alone is not enough. Anyone can build a list of agents.

The harder problem is invocation. Agent runtimes expose similar primitives — agents, sessions, events, tools — but they do not expose them through the same APIs. So if you want one place to actually use these agents, not just list them, the control plane has to manage agent runtimes, schedules, memory, and sessions.

This is the same pattern LiteLLM saw with models. Companies did not just need a catalog of models. They needed one interface to call them. The only change, is that the primitive is now the agent session, not the model call.

The Stack of the Future​

The important shift is that the gateway is no longer just routing model calls. It is routing agent work.

With LLMs, the stack became:

• Models: GPT, Claude, Gemini, Llama
• Inference providers: OpenAI, Anthropic, Bedrock, Vertex, Azure, vLLM
• Gateway: routing, fallbacks, logging, spend tracking, auth, billing
• Applications: copilots, workflows, internal tools, products

With agents, we think the stack becomes:

• Models: Claude, GPT, Gemini, open-source models
• Harnesses: Claude Code, Codex, OpenCode, Hermes, DeepAgents
• Agent runtimes: Claude Managed Agents, Bedrock AgentCore, Gemini Enterprise Agent Platform, self-hosted runtimes
• Agent control plane: multi-runtime platform where teams manage agent runtimes, schedules, memory, and sessions.
• Applications: coding agents, support agents, data agents, security agents

Why companies will need this​

At LiteLLM, we are already seeing our team work across multiple agent runtimes. Some people are building on Claude Managed Agents, others are on N8N or Cursor.

This fragmentation makes it hard for agents built on these platforms to be shareable, and everyone to benefit from the work done so far.

By having the agents live in 1 place, everyone can leverage these agents - even if the PR Babysitter Agent was written in Claude Managed Agents, which not everyone has direct access to.

That is the control plane problem.

This is also why we think the AI Gateway moves up the stack. The gateway starts by managing model calls. But as agents become the dominant use-case for AI, the gateway has to manage agent sessions too.

What we are building​

LiteLLM Agent Platform is our experiment in this direction.

LiteLLM Agent Platform is a Rust-based AI Gateway and Agent Control Plane. The goal is to let teams register, invoke, observe, and govern agents across multiple runtimes.

We are starting with coding agents because the need is obvious. They are long-running, stateful, tool-heavy, and expensive enough to require real infrastructure.

We are already seeing early users resonate with this pattern. Some companies want LAP to act as a central control plane for agents built by different teams on different runtimes. For example, one team might build an agent on Elastic’s runtime to analyze Kibana logs, but the company may want to expose that agent internally through a common gateway.

This is the architecture we believe is coming: models become interchangeable, harnesses become specialized, runtimes become managed, and the gateway becomes the control plane for agent work.

If this matches what you are seeing, we would love feedback on LiteLLM Agent Platform:

https://github.com/LiteLLM-Labs/litellm-agent-platform

Frequently Asked Questions​

Is LiteLLM building a second product?​

No. LAP is an experimental project. The goal is to learn quickly and bring the right pieces into LiteLLM over time.

Is LAP production-ready?​

No. LAP is pre-v0. APIs may change as we work with early users and contributors.

If you want to contribute, file an issue or join our Discord:

https://discord.gg/Nkxw3rm3EE

What is ASSERT?​

ASSERT is Microsoft's open-source framework for policy-driven agent evaluation, built on a proven Microsoft Research approach. ASSERT takes your organizational policies and requirements as input, systematically generates targeted evaluation scenarios, and surfaces safety and quality defects before they reach production.

Why this matters​

As teams ship agents into production, the gap between "it works in a demo" and "it behaves under our policies" is where real risk lives. ASSERT closes that gap by turning your written policies into concrete, testable evaluation scenarios — and now those evaluations run against any of the 100+ LLM providers LiteLLM supports, through a single unified interface.

How it works with LiteLLM​

• Bring your policies — ASSERT ingests your organizational policies and requirements.
• Generate scenarios — ASSERT systematically produces targeted evaluation scenarios.
• Run through LiteLLM — evaluate any model behind the LiteLLM Gateway with consistent auth, logging, and cost tracking.
• Surface defects early — catch safety and quality issues before they reach production.

Get started​

ASSERT is open source. Point it at your LiteLLM Gateway endpoint and start evaluating your agents against your own policies today.

• Set up the LiteLLM Gateway — get a gateway endpoint running in minutes.
• Microsoft ASSERT on GitHub — install ASSERT and run it against your gateway.

It exposes harnesses in a unified Claude Agents SDK spec. This means that if you wrote your app with the Claude Agents SDK, and want to try another harness (Pi AI, Hermes, Codex, OpenCode), you can do so without rewriting your code.

Today, it supports 3 harnesses - Claude Code, Codex, and Pi AI. Please file an issue here, if you want us to add another harness.

Here's how it works:

TypeScript Example

import { query } from "@lite-harness/sdk";

const prompt = "Fix the failing test";

// Claude Code harness
for await (const message of query({
  prompt,
  options: { harness: "claude-code", model: "claude-opus-4-8" },
})) {
  console.log(message);
}

// Codex harness
for await (const message of query({
  prompt,
  options: { harness: "codex", model: "gpt-5.5" },
})) {
  console.log(message);
}

Python Example

from lite_harness import query, AgentOptions

prompt = "Fix the failing test"

# Claude Code harness
async for message in query(
    prompt=prompt,
    options=AgentOptions(harness="claude-code", model="claude-opus-4-8"),
):
    print(message)

# Codex harness
async for message in query(
    prompt=prompt,
    options=AgentOptions(harness="codex", model="gpt-5.5"),
):
    print(message)

LiteLLM AI Gateway​

Lite-Harness supports proxy'ing harnesses via LiteLLM AI Gateway. This enables easy model swapping, cost controls and logging.

Point Lite-Harness at your gateway by setting two environment variables:

export LITELLM_API_BASE=https://litellm.your-company.com/v1
export LITELLM_API_KEY=sk-litellm-...

Then call as usual — every underlying model request routes through the gateway:

from lite_harness import query, AgentOptions

prompt = "Fix the failing test"

# Claude Code harness
async for message in query(
    prompt=prompt,
    options=AgentOptions(harness="claude-code", model="claude-opus-4-8"),
):
    print(message)

# Codex harness
async for message in query(
    prompt=prompt,
    options=AgentOptions(harness="codex", model="gpt-5.5"),
):
    print(message)

Frequently Asked Questions​

Do I have to use the LiteLLM AI Gateway?​

No. lite-harness works standalone — point it at provider APIs with native keys. AI Gateway integration is opt-in for teams that want central key management, budgets, fallbacks, and a single audit log across every model call.

Does swapping harnesses change agent behavior?​

Yes — that's the point. Each harness keeps its native loop, tool-calling semantics, and prompt format. lite-harness unifies how you invoke them, not how they run internally. Run the same prompt across all three to see which combo lands the task best.

Is this ready for production?​

lite-harness is an early, experimental project. This is in public beta. Please join our discord, to help design it to your preference.

Is this available in LiteLLM OSS?​

Yes. lite-harness is MIT-licensed at github.com/LiteLLM-Labs/lite-harness. LiteLLM Enterprise adds SSO/SCIM, air-gapped deployment, 24/7 SLA, and advanced guardrails on top of the AI Gateway it pairs with.

Recommended Reading​

• LiteLLM AI Gateway — full feature overview
• LiteLLM Managed Agents Platform — Alpha
• Load balancing and routing across 100+ LLM providers

The conditions could allow unauthenticated access to protected management routes when the proxy listener was reachable with an arbitrary Host header.

No LiteLLM Cloud customers were affected. The update was deployed across all LiteLLM Cloud environments - backported to the release lines in use - ahead of this publication.

• Addressed in: v1.84.0
• Recommended: the latest release; follow-up path-handling hardening was backported in v1.84.3, v1.85.2, and v1.86.2
• Action: upgrade to v1.84.0 or later. No configuration change is required.

More info on the advisory is here: https://github.com/BerriAI/litellm/security/advisories/GHSA-4xpc-pv4p-pm3w. CVE: https://www.cve.org/CVERecord?id=CVE-2026-48710.

TL;DR​

• A crafted Host header could make the proxy's auth gate evaluate a different route from the one it served, allowing potential unauthenticated access to protected management routes.
• The update shipped in v1.84.0. Follow-up path-handling hardening was backported in v1.84.3, v1.85.2, and v1.86.2; upgrading to the latest release is recommended.
• Potential bypass requires reaching the proxy listener with an arbitrary Host header. Fronting the proxy with infrastructure that validates or normalizes Host reduces potential for bypass depending on configuration, but is not a comprehensive substitute for upgrading.
• No LiteLLM Cloud customers were affected.

Summary​

The proxy's auth layer derived the effective route from request.url.path in litellm/proxy/auth/auth_utils.py::get_request_route(), which Starlette reconstructs from the Host header. A crafted Host header could therefore make the auth gate evaluate a different route from the one FastAPI actually dispatched, causing a protected management route to be treated as public.

Potential bypass requires an actor to reach the proxy listener with an arbitrary Host header. Fronting the proxy with infrastructure that validates or normalizes the Host header reduces potential for bypass, though whether it fully blocks the bypass depends on the specific configuration. The LiteLLM Python SDK is not affected; only the proxy server is in limited scope.

Additional hardening​

The primary update in v1.84.0 addressed the reported potential for bypass by deriving the request route from the ASGI scope path rather than the Host-reconstructed URL. As additional follow-up, we audited every other location in the proxy that derived a route from the request URL and moved them onto the same hardened resolution. This closes the long tail of the potential for bypass and was backported across the maintained release lines in v1.84.3, v1.85.2, v1.86.2, and v1.83.10-stable.patch.3. We recommend upgrading to one of these releases for comprehensive mitigation.

Am I affected?​

You are potentially affected only if all of the following are true:

• You run the LiteLLM proxy server (not just the Python SDK).
• You are on a version earlier than v1.84.0.
• The proxy listener is reachable by untrusted clients.

You are not remotely open to potential bypass if the proxy listener is not reachable by untrusted clients — for example, it is bound to a private network or sits behind a gateway that requires its own authentication.

Fronting the proxy with infrastructure that validates or normalizes the Host header (a CDN/WAF, a reverse proxy with server_name allowlists, or a host-based load balancer) reduces potential for bypass, but whether it fully mitigates against potential bypass depends on the configuration.

What to do​

1. Upgrade to v1.84.0 or later. Upgrading to the latest release is recommended, which includes the follow-up hardening backported in v1.84.3, v1.85.2, and v1.86.2.
2. If your proxy was reachable from an untrusted network on an affected version, rotate any API keys created during the exposure window and review your management audit logs for unexpected key, user, or settings changes.

Mitigations​

If you cannot upgrade immediately, to better mitigate the potential for bypass, we recommend placing the proxy behind an upstream component that validates or normalizes the Host header before forwarding:

• a CDN or WAF (e.g. Cloudflare),
• a reverse proxy with explicit server_name allowlists (nginx, Caddy, Traefik),
• a cloud load balancer with host-based routing rules,

or otherwise restrict network access to the proxy listener. Note this is a per-deployment property: a reverse proxy that forwards the client Host unchanged (e.g. nginx proxy_set_header Host $host;) may not comprehensively protect your use from this potential. Treat upgrading as the elimination of any potential for bypass and edge filtering only as a stopgap.