---
title: "/responses"
url: "/en/docs/response_api"
canonical_url: "https://docs.litellm.ai/en/docs/response_api"
type: "docs"
summary: "LiteLLM provides an endpoint in the spec of OpenAI's /responses API"
related:
  - "/en/docs/embedding/supported_embedding"
  - "/en/docs/text_completion"
---
# /responses

> Index of all LiteLLM docs: https://docs.litellm.ai/llms.txt


LiteLLM provides an endpoint in the spec of [OpenAI's `/responses` API](https://platform.openai.com/docs/api-reference/responses)

Requests to /chat/completions may be bridged here automatically when the provider lacks support for that endpoint. The model’s default `mode` determines how bridging works.(see `model_prices_and_context_window`) 

| Feature | Supported | Notes |
|---------|-----------|--------|
| Cost Tracking | ✅ | Works with all supported models |
| Logging | ✅ | Works across all integrations |
| End-user Tracking | ✅ | |
| Streaming | ✅ | |
| WebSocket Mode | ✅ | Lower-latency persistent connections for all providers |
| Image Generation Streaming | ✅ | Progressive image generation with partial images (1-3) |
| Fallbacks | ✅ | Works between supported models |
| Loadbalancing | ✅ | Works between supported models |
| Guardrails | ✅ | Applies to input and output text (non-streaming only) |
| Supported operations | Create a response, Get a response, Delete a response | |
| Supported LiteLLM Versions | 1.63.8+ | |
| Supported LLM providers | **All LiteLLM supported providers** | `openai`, `anthropic`, `bedrock`, `vertex_ai`, `gemini`, `azure`, `azure_ai` etc. |

## Usage

### LiteLLM Python SDK

**OpenAI**

#### Non-streaming
```python showLineNumbers title="OpenAI Non-streaming Response"
import litellm

# Non-streaming response
response = litellm.responses(
    model="openai/gpt-5.6-terra",
    input="Tell me a three sentence bedtime story about a unicorn.",
    max_output_tokens=100
)

print(response)
```

#### Response Format (OpenAI Responses API Format)

```json
{
    "id": "resp_abc123",
    "object": "response",
    "created_at": 1734366691,
    "status": "completed",
    "model": "gpt-5.6-terra",
    "output": [
        {
            "type": "message",
            "id": "msg_abc123",
            "status": "completed",
            "role": "assistant",
            "content": [
                {
                    "type": "output_text",
                    "text": "Once upon a time, a little unicorn named Stardust lived in a magical meadow where flowers sang lullabies. One night, she discovered that her horn could paint dreams across the sky, and she spent the evening creating the most beautiful aurora for all the forest creatures to enjoy. As the animals drifted off to sleep beneath her shimmering lights, Stardust curled up on a cloud of moonbeams, happy to have shared her magic with her friends.",
                    "annotations": []
                }
            ]
        }
    ],
    "usage": {
        "input_tokens": 18,
        "output_tokens": 98,
        "total_tokens": 116
    }
}
```

#### Streaming
```python showLineNumbers title="OpenAI Streaming Response"
import litellm

# Streaming response
response = litellm.responses(
    model="openai/gpt-5.6-terra",
    input="Tell me a three sentence bedtime story about a unicorn.",
    stream=True
)

for event in response:
    print(event)
```

#### Image Generation with Streaming
```python showLineNumbers title="OpenAI Streaming Image Generation"
import litellm
import base64

# Streaming image generation with partial images
stream = litellm.responses(
    model="gpt-5.6-terra",  # Use an actual image generation model
    input="Generate a gorgeous image of a river made of white owl feathers",
    stream=True,
    tools=[{"type": "image_generation", "partial_images": 2}],

)

for event in stream:
    if event.type == "response.image_generation_call.partial_image":
        idx = event.partial_image_index
        image_base64 = event.partial_image_b64
        image_bytes = base64.b64decode(image_base64)
        with open(f"river{idx}.png", "wb") as f:
            f.write(image_bytes)
```

#### Image Generation (Non-streaming)

Image generation is supported for models that generate images. Generated images are returned in the `output` array with `type: "image_generation_call"`.

**Gemini (Google AI Studio):**
```python showLineNumbers title="Gemini Image Generation"
import litellm
import base64

# Gemini image generation models don't require tools parameter
response = litellm.responses(
    model="gemini/gemini-2.5-flash-image",
    input="Generate a cute cat playing with yarn"
)

# Access generated images from output
for item in response.output:
    if item.type == "image_generation_call":
        # item.result contains pure base64 (no data: prefix)
        image_bytes = base64.b64decode(item.result)

        # Save the image
        with open(f"generated_{item.id}.png", "wb") as f:
            f.write(image_bytes)

print(f"Image saved: generated_{response.output[0].id}.png")
```

**OpenAI:**
```python showLineNumbers title="OpenAI Image Generation"
import litellm
import base64

# OpenAI models require tools parameter for image generation
response = litellm.responses(
    model="openai/gpt-5.6-terra",
    input="Generate a futuristic city at sunset",
    tools=[{"type": "image_generation"}]
)

# Access generated images from output
for item in response.output:
    if item.type == "image_generation_call":
        image_bytes = base64.b64decode(item.result)
        with open(f"generated_{item.id}.png", "wb") as f:
            f.write(image_bytes)
```

**Response Format:**

When image generation is successful, the response contains:

```json
{
  "id": "resp_abc123",
  "status": "completed",
  "output": [
    {
      "type": "image_generation_call",
      "id": "resp_abc123_img_0",
      "status": "completed",
      "result": "iVBORw0KGgo..."  // Pure base64 string (no data: prefix)
    }
  ]
}
```

**Supported Models:**

| Provider | Models | Requires `tools` Parameter |
|----------|--------|---------------------------|
| Google AI Studio | `gemini/gemini-2.5-flash-image` | ❌ No |
| Vertex AI | `vertex_ai/gemini-2.5-flash-image-preview` | ❌ No |
| OpenAI | `gpt-4o`, `gpt-4o-mini`, `gpt-4.1`, `gpt-4.1-mini`, `gpt-4.1-nano`, `o3` | ✅ Yes |
| AWS Bedrock | Stability AI, Amazon Nova Canvas models | Model-specific |
| Fal AI | Various image generation models | Check model docs |

**Note:** The `result` field contains pure base64-encoded image data without the `data:image/png;base64,` prefix. You must decode it with `base64.b64decode()` before saving.

#### GET a Response
```python showLineNumbers title="Get Response by ID"
import litellm

# First, create a response
response = litellm.responses(
    model="openai/gpt-5.6-terra",
    input="Tell me a three sentence bedtime story about a unicorn.",
    max_output_tokens=100
)

# Get the response ID
response_id = response.id

# Retrieve the response by ID
retrieved_response = litellm.get_responses(
    response_id=response_id
)

print(retrieved_response)

# For async usage
# retrieved_response = await litellm.aget_responses(response_id=response_id)
```

#### CANCEL a Response
You can cancel an in-progress response (if supported by the provider):

```python showLineNumbers title="Cancel Response by ID"
import litellm

# First, create a response
response = litellm.responses(
    model="openai/gpt-5.6-terra",
    input="Tell me a three sentence bedtime story about a unicorn.",
    max_output_tokens=100
)

# Get the response ID
response_id = response.id

# Cancel the response by ID
cancel_response = litellm.cancel_responses(
    response_id=response_id
)

print(cancel_response)

# For async usage
# cancel_response = await litellm.acancel_responses(response_id=response_id)
```

**REST API:**
```bash
curl -X POST http://localhost:4000/v1/responses/response_id/cancel \
    -H "Authorization: Bearer $LITELLM_API_KEY"
```

This will attempt to cancel the in-progress response with the given ID.
**Note:** Not all providers support response cancellation. If unsupported, an error will be raised.

#### DELETE a Response
```python showLineNumbers title="Delete Response by ID"
import litellm

# First, create a response
response = litellm.responses(
    model="openai/gpt-5.6-terra",
    input="Tell me a three sentence bedtime story about a unicorn.",
    max_output_tokens=100
)

# Get the response ID
response_id = response.id

# Delete the response by ID
delete_response = litellm.delete_responses(
    response_id=response_id
)

print(delete_response)

# For async usage
# delete_response = await litellm.adelete_responses(response_id=response_id)
```

**Anthropic**

#### Non-streaming
```python showLineNumbers title="Anthropic Non-streaming Response"
import litellm
import os

# Set API key
os.environ["ANTHROPIC_API_KEY"] = "your-anthropic-api-key"

# Non-streaming response
response = litellm.responses(
    model="anthropic/claude-sonnet-5",
    input="Tell me a three sentence bedtime story about a unicorn.",
    max_output_tokens=100
)

print(response)
```

#### Streaming
```python showLineNumbers title="Anthropic Streaming Response"
import litellm
import os

# Set API key
os.environ["ANTHROPIC_API_KEY"] = "your-anthropic-api-key"

# Streaming response
response = litellm.responses(
    model="anthropic/claude-sonnet-5",
    input="Tell me a three sentence bedtime story about a unicorn.",
    stream=True
)

for event in response:
    print(event)
```

**Vertex AI**

#### Non-streaming
```python showLineNumbers title="Vertex AI Non-streaming Response"
import litellm
import os

# Set credentials - Vertex AI uses application default credentials
# Run 'gcloud auth application-default login' to authenticate
os.environ["VERTEXAI_PROJECT"] = "your-gcp-project-id"
os.environ["VERTEXAI_LOCATION"] = "us-central1"

# Non-streaming response
response = litellm.responses(
    model="vertex_ai/gemini-3.1-pro-preview",
    input="Tell me a three sentence bedtime story about a unicorn.",
    max_output_tokens=100
)

print(response)
```

#### Streaming
```python showLineNumbers title="Vertex AI Streaming Response"
import litellm
import os

# Set credentials - Vertex AI uses application default credentials
# Run 'gcloud auth application-default login' to authenticate
os.environ["VERTEXAI_PROJECT"] = "your-gcp-project-id"
os.environ["VERTEXAI_LOCATION"] = "us-central1"

# Streaming response
response = litellm.responses(
    model="vertex_ai/gemini-3.1-pro-preview",
    input="Tell me a three sentence bedtime story about a unicorn.",
    stream=True
)

for event in response:
    print(event)
```

**AWS Bedrock**

#### Non-streaming
```python showLineNumbers title="AWS Bedrock Non-streaming Response"
import litellm
import os

# Set AWS credentials
os.environ["AWS_ACCESS_KEY_ID"] = "your-access-key-id"
os.environ["AWS_SECRET_ACCESS_KEY"] = "your-secret-access-key"
os.environ["AWS_REGION_NAME"] = "us-west-2"  # or your AWS region

# Non-streaming response
response = litellm.responses(
    model="bedrock/us.anthropic.claude-sonnet-5",
    input="Tell me a three sentence bedtime story about a unicorn.",
    max_output_tokens=100
)

print(response)
```

#### Streaming
```python showLineNumbers title="AWS Bedrock Streaming Response"
import litellm
import os

# Set AWS credentials
os.environ["AWS_ACCESS_KEY_ID"] = "your-access-key-id"
os.environ["AWS_SECRET_ACCESS_KEY"] = "your-secret-access-key"
os.environ["AWS_REGION_NAME"] = "us-west-2"  # or your AWS region

# Streaming response
response = litellm.responses(
    model="bedrock/us.anthropic.claude-sonnet-5",
    input="Tell me a three sentence bedtime story about a unicorn.",
    stream=True
)

for event in response:
    print(event)
```

**Google AI Studio**

#### Non-streaming
```python showLineNumbers title="Google AI Studio Non-streaming Response"
import litellm
import os

# Set API key for Google AI Studio
os.environ["GEMINI_API_KEY"] = "your-gemini-api-key"

# Non-streaming response
response = litellm.responses(
    model="gemini/gemini-3.8-flash",
    input="Tell me a three sentence bedtime story about a unicorn.",
    max_output_tokens=100
)

print(response)
```

#### Streaming
```python showLineNumbers title="Google AI Studio Streaming Response"
import litellm
import os

# Set API key for Google AI Studio
os.environ["GEMINI_API_KEY"] = "your-gemini-api-key"

# Streaming response
response = litellm.responses(
    model="gemini/gemini-3.8-flash",
    input="Tell me a three sentence bedtime story about a unicorn.",
    stream=True
)

for event in response:
    print(event)
```

### LiteLLM Proxy with OpenAI SDK

First, set up and start your LiteLLM proxy server.

```bash title="Start LiteLLM Proxy Server"
litellm --config /path/to/config.yaml

# RUNNING on http://0.0.0.0:4000
```

**OpenAI**

First, add this to your litellm proxy config.yaml:
```yaml showLineNumbers title="OpenAI Proxy Configuration"
model_list:
  - model_name: openai/gpt-5.6-terra
    litellm_params:
      model: openai/gpt-5.6-terra
      api_key: os.environ/OPENAI_API_KEY
```

#### Non-streaming
```python showLineNumbers title="OpenAI Proxy Non-streaming Response"
from openai import OpenAI

# Initialize client with your proxy URL
client = OpenAI(
    base_url="http://localhost:4000",  # Your proxy URL
    api_key="your-api-key"             # Your proxy API key
)

# Non-streaming response
response = client.responses.create(
    model="openai/gpt-5.6-terra",
    input="Tell me a three sentence bedtime story about a unicorn."
)

print(response)
```

#### Streaming
```python showLineNumbers title="OpenAI Proxy Streaming Response"
from openai import OpenAI

# Initialize client with your proxy URL
client = OpenAI(
    base_url="http://localhost:4000",  # Your proxy URL
    api_key="your-api-key"             # Your proxy API key
)

# Streaming response
response = client.responses.create(
    model="openai/gpt-5.6-terra",
    input="Tell me a three sentence bedtime story about a unicorn.",
    stream=True
)

for event in response:
    print(event)
```

#### Image Generation with Streaming
```python showLineNumbers title="OpenAI Proxy Streaming Image Generation"
from openai import OpenAI
import base64

client = OpenAI(api_key="sk-<your-litellm-api-key>", base_url="http://localhost:4000")

stream = client.responses.create(
    model="gpt-5.6-terra",
    input="Draw a gorgeous image of a river made of white owl feathers, snaking its way through a serene winter landscape",
    stream=True,
    tools=[{"type": "image_generation", "partial_images": 2}],
)

for event in stream:
    print(f"event: {event}")
    if event.type == "response.image_generation_call.partial_image":
        idx = event.partial_image_index
        image_base64 = event.partial_image_b64
        image_bytes = base64.b64decode(image_base64)
        with open(f"river{idx}.png", "wb") as f:
            f.write(image_bytes)

```

#### GET a Response
```python showLineNumbers title="Get Response by ID with OpenAI SDK"
from openai import OpenAI

# Initialize client with your proxy URL
client = OpenAI(
    base_url="http://localhost:4000",  # Your proxy URL
    api_key="your-api-key"             # Your proxy API key
)

# First, create a response
response = client.responses.create(
    model="openai/gpt-5.6-terra",
    input="Tell me a three sentence bedtime story about a unicorn."
)

# Get the response ID
response_id = response.id

# Retrieve the response by ID
retrieved_response = client.responses.retrieve(response_id)

print(retrieved_response)
```

#### DELETE a Response
```python showLineNumbers title="Delete Response by ID with OpenAI SDK"
from openai import OpenAI

# Initialize client with your proxy URL
client = OpenAI(
    base_url="http://localhost:4000",  # Your proxy URL
    api_key="your-api-key"             # Your proxy API key
)

# First, create a response
response = client.responses.create(
    model="openai/gpt-5.6-terra",
    input="Tell me a three sentence bedtime story about a unicorn."
)

# Get the response ID
response_id = response.id

# Delete the response by ID
delete_response = client.responses.delete(response_id)

print(delete_response)
```

**Anthropic**

First, add this to your litellm proxy config.yaml:
```yaml showLineNumbers title="Anthropic Proxy Configuration"
model_list:
  - model_name: anthropic/claude-sonnet-5
    litellm_params:
      model: anthropic/claude-sonnet-5
      api_key: os.environ/ANTHROPIC_API_KEY
```

#### Non-streaming
```python showLineNumbers title="Anthropic Proxy Non-streaming Response"
from openai import OpenAI

# Initialize client with your proxy URL
client = OpenAI(
    base_url="http://localhost:4000",  # Your proxy URL
    api_key="your-api-key"             # Your proxy API key
)

# Non-streaming response
response = client.responses.create(
    model="anthropic/claude-sonnet-5",
    input="Tell me a three sentence bedtime story about a unicorn."
)

print(response)
```

#### Streaming
```python showLineNumbers title="Anthropic Proxy Streaming Response"
from openai import OpenAI

# Initialize client with your proxy URL
client = OpenAI(
    base_url="http://localhost:4000",  # Your proxy URL
    api_key="your-api-key"             # Your proxy API key
)

# Streaming response
response = client.responses.create(
    model="anthropic/claude-sonnet-5",
    input="Tell me a three sentence bedtime story about a unicorn.",
    stream=True
)

for event in response:
    print(event)
```

**Vertex AI**

First, add this to your litellm proxy config.yaml:
```yaml showLineNumbers title="Vertex AI Proxy Configuration"
model_list:
  - model_name: vertex_ai/gemini-3.1-pro-preview
    litellm_params:
      model: vertex_ai/gemini-3.1-pro-preview
      vertex_project: your-gcp-project-id
      vertex_location: us-central1
```

#### Non-streaming
```python showLineNumbers title="Vertex AI Proxy Non-streaming Response"
from openai import OpenAI

# Initialize client with your proxy URL
client = OpenAI(
    base_url="http://localhost:4000",  # Your proxy URL
    api_key="your-api-key"             # Your proxy API key
)

# Non-streaming response
response = client.responses.create(
    model="vertex_ai/gemini-3.1-pro-preview",
    input="Tell me a three sentence bedtime story about a unicorn."
)

print(response)
```

#### Streaming
```python showLineNumbers title="Vertex AI Proxy Streaming Response"
from openai import OpenAI

# Initialize client with your proxy URL
client = OpenAI(
    base_url="http://localhost:4000",  # Your proxy URL
    api_key="your-api-key"             # Your proxy API key
)

# Streaming response
response = client.responses.create(
    model="vertex_ai/gemini-3.1-pro-preview",
    input="Tell me a three sentence bedtime story about a unicorn.",
    stream=True
)

for event in response:
    print(event)
```

**AWS Bedrock**

First, add this to your litellm proxy config.yaml:
```yaml showLineNumbers title="AWS Bedrock Proxy Configuration"
model_list:
  - model_name: bedrock/us.anthropic.claude-sonnet-5
    litellm_params:
      model: bedrock/us.anthropic.claude-sonnet-5
      aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
      aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
      aws_region_name: us-west-2
```

#### Non-streaming
```python showLineNumbers title="AWS Bedrock Proxy Non-streaming Response"
from openai import OpenAI

# Initialize client with your proxy URL
client = OpenAI(
    base_url="http://localhost:4000",  # Your proxy URL
    api_key="your-api-key"             # Your proxy API key
)

# Non-streaming response
response = client.responses.create(
    model="bedrock/us.anthropic.claude-sonnet-5",
    input="Tell me a three sentence bedtime story about a unicorn."
)

print(response)
```

#### Streaming
```python showLineNumbers title="AWS Bedrock Proxy Streaming Response"
from openai import OpenAI

# Initialize client with your proxy URL
client = OpenAI(
    base_url="http://localhost:4000",  # Your proxy URL
    api_key="your-api-key"             # Your proxy API key
)

# Streaming response
response = client.responses.create(
    model="bedrock/us.anthropic.claude-sonnet-5",
    input="Tell me a three sentence bedtime story about a unicorn.",
    stream=True
)

for event in response:
    print(event)
```

**Google AI Studio**

First, add this to your litellm proxy config.yaml:
```yaml showLineNumbers title="Google AI Studio Proxy Configuration"
model_list:
  - model_name: gemini/gemini-3.8-flash
    litellm_params:
      model: gemini/gemini-3.8-flash
      api_key: os.environ/GEMINI_API_KEY
```

#### Non-streaming
```python showLineNumbers title="Google AI Studio Proxy Non-streaming Response"
from openai import OpenAI

# Initialize client with your proxy URL
client = OpenAI(
    base_url="http://localhost:4000",  # Your proxy URL
    api_key="your-api-key"             # Your proxy API key
)

# Non-streaming response
response = client.responses.create(
    model="gemini/gemini-3.8-flash",
    input="Tell me a three sentence bedtime story about a unicorn."
)

print(response)
```

#### Streaming
```python showLineNumbers title="Google AI Studio Proxy Streaming Response"
from openai import OpenAI

# Initialize client with your proxy URL
client = OpenAI(
    base_url="http://localhost:4000",  # Your proxy URL
    api_key="your-api-key"             # Your proxy API key
)

# Streaming response
response = client.responses.create(
    model="gemini/gemini-3.8-flash",
    input="Tell me a three sentence bedtime story about a unicorn.",
    stream=True
)

for event in response:
    print(event)
```

## WebSocket Mode

The Responses API supports **WebSocket mode** for lower-latency, persistent connections ideal for agentic workflows. WebSocket mode works with **all LiteLLM providers**, not just those with native WebSocket support.

### Architecture

LiteLLM provides two WebSocket modes:

1. **Native WebSocket**: Direct `wss://` connection to providers that support it (OpenAI, Azure)
2. **Managed WebSocket**: HTTP streaming over WebSocket for all other providers (Anthropic, Gemini, Bedrock, etc.)

The system automatically selects the appropriate mode based on provider capabilities.

### Usage

**Python (websocket-client)**

```python showLineNumbers title="WebSocket with Python"
import json
from websocket import create_connection  # uv add websocket-client

# Connect to LiteLLM proxy WebSocket endpoint
ws = create_connection(
    "ws://localhost:4000/v1/responses?model=gemini-3.8-flash",
    header=["Authorization: Bearer sk-<your-litellm-api-key>"]
)

try:
    # Send initial message
    ws.send(json.dumps({
        "type": "response.create",
        "model": "gemini-3.8-flash",
        "store": True,
        "input": [{
            "type": "message",
            "role": "user",
            "content": [{"type": "input_text", "text": "My favorite color is blue."}]
        }]
    }))
    
    # Collect response events
    response_id = None
    while True:
        event = json.loads(ws.recv())
        print(f"Event: {event['type']}")
        
        if event["type"] == "response.completed":
            response_id = event["response"]["id"]
            break
        elif event["type"] == "response.output_text.delta":
            print(f"Text: {event.get('delta', '')}", end="", flush=True)
    
    print(f"\nResponse ID: {response_id}")
    
    # Send follow-up with previous_response_id for multi-turn
    ws.send(json.dumps({
        "type": "response.create",
        "model": "gemini-3.8-flash",
        "previous_response_id": response_id,
        "input": [{
            "type": "message",
            "role": "user",
            "content": [{"type": "input_text", "text": "What is my favorite color?"}]
        }]
    }))
    
    # Collect follow-up response
    while True:
        event = json.loads(ws.recv())
        if event["type"] == "response.completed":
            break
        elif event["type"] == "response.output_text.delta":
            print(event.get("delta", ""), end="", flush=True)
            
finally:
    ws.close()
```

**JavaScript (ws)**

```javascript showLineNumbers title="WebSocket with JavaScript"
const WebSocket = require('ws'); // npm install ws

const ws = new WebSocket(
    'ws://localhost:4000/v1/responses?model=gemini-3.8-flash',
    {
        headers: {
            'Authorization': 'Bearer sk-<your-litellm-api-key>'
        }
    }
);

ws.on('open', () => {
    // Send initial message
    ws.send(JSON.stringify({
        type: 'response.create',
        model: 'gemini-3.8-flash',
        store: true,
        input: [{
            type: 'message',
            role: 'user',
            content: [{ type: 'input_text', text: 'My favorite color is blue.' }]
        }]
    }));
});

let responseId = null;

ws.on('message', (data) => {
    const event = JSON.parse(data.toString());
    console.log(`Event: ${event.type}`);
    
    if (event.type === 'response.completed') {
        responseId = event.response.id;
        console.log(`Response ID: ${responseId}`);
        
        // Send follow-up
        ws.send(JSON.stringify({
            type: 'response.create',
            model: 'gemini-3.8-flash',
            previous_response_id: responseId,
            input: [{
                type: 'message',
                role: 'user',
                content: [{ type: 'input_text', text: 'What is my favorite color?' }]
            }]
        }));
    } else if (event.type === 'response.output_text.delta') {
        process.stdout.write(event.delta || '');
    }
});

ws.on('error', (error) => {
    console.error('WebSocket error:', error);
});
```

**curl (websocat)**

```bash showLineNumbers title="WebSocket with websocat"
# Install websocat: brew install websocat (macOS) or cargo install websocat

# Connect to WebSocket endpoint
websocat "ws://localhost:4000/v1/responses?model=gemini-3.8-flash" \
  -H="Authorization: Bearer $LITELLM_API_KEY"

# Then send JSON events (paste and press Enter):
{"type":"response.create","model":"gemini-3.8-flash","input":[{"type":"message","role":"user","content":[{"type":"input_text","text":"Hello!"}]}]}

# You'll receive streaming events back:
# {"type":"response.created",...}
# {"type":"response.in_progress",...}
# {"type":"response.output_text.delta","delta":"Hello",...}
# {"type":"response.completed",...}
```

### Event Types

WebSocket connections receive Server-Sent Events (SSE) formatted as JSON:

| Event Type | Description |
|------------|-------------|
| `response.created` | Response generation started |
| `response.in_progress` | Response is being generated |
| `response.output_item.added` | New output item (message, tool call, etc.) added |
| `response.output_text.delta` | Incremental text chunk |
| `response.output_text.done` | Text output completed |
| `response.content_part.done` | Content part completed |
| `response.output_item.done` | Output item completed |
| `response.completed` | Full response completed successfully |
| `response.failed` | Response generation failed |
| `response.incomplete` | Response incomplete (e.g., max tokens reached) |
| `error` | Error occurred |

### Multi-Turn Conversations

Use `previous_response_id` to maintain conversation context across multiple WebSocket messages:

```python showLineNumbers title="Multi-turn WebSocket Conversation"
# Turn 1
ws.send(json.dumps({
    "type": "response.create",
    "model": "gemini-3.8-flash",
    "store": True,  # Required for multi-turn
    "input": [{"type": "message", "role": "user", "content": [{"type": "input_text", "text": "Hello"}]}]
}))

# ... collect events and get response_id from response.completed event ...

# Turn 2 - reference previous response
ws.send(json.dumps({
    "type": "response.create",
    "model": "gemini-3.8-flash",
    "previous_response_id": response_id,  # Links to previous turn
    "input": [{"type": "message", "role": "user", "content": [{"type": "input_text", "text": "Continue"}]}]
}))
```

### Provider Support

| Provider | WebSocket Mode | Notes |
|----------|----------------|-------|
| OpenAI | Native | Direct `wss://` connection to OpenAI |
| Azure OpenAI | Native | Direct `wss://` connection to Azure |
| Anthropic | Managed | HTTP streaming over WebSocket |
| Google AI Studio (Gemini) | Managed | HTTP streaming over WebSocket |
| Vertex AI | Managed | HTTP streaming over WebSocket |
| AWS Bedrock | Managed | HTTP streaming over WebSocket |
| All other providers | Managed | HTTP streaming over WebSocket |

**Note**: Both native and managed modes provide the same event stream format. The difference is transparent to clients.

### Configuration

No special configuration needed. WebSocket mode is automatically available on the `/v1/responses` endpoint when accessed via WebSocket protocol (`ws://` or `wss://`).

For LiteLLM Proxy, ensure your models are configured normally:

```yaml showLineNumbers title="config.yaml"
model_list:
  - model_name: gemini-3.8-flash
    litellm_params:
      model: gemini/gemini-3.8-flash
      api_key: os.environ/GEMINI_API_KEY
  
  - model_name: gpt-5.6-terra
    litellm_params:
      model: openai/gpt-5.6-terra
      api_key: os.environ/OPENAI_API_KEY
```

Both models will automatically support WebSocket mode at `ws://localhost:4000/v1/responses`.

## Response ID Security

By default, LiteLLM Proxy prevents users from accessing other users' response IDs.

This is done by encrypting the response ID with the user ID, enabling users to only access their own response IDs.

Trying to access someone else's response ID returns 403:

```json
{
  "error": {
    "message": "Forbidden. The response id is not associated with the user, who this key belongs to.",
    "code": 403
  }
}
```

To disable this, set `disable_responses_id_security: true`:

```yaml
general_settings:
  disable_responses_id_security: true
```

This allows any user to access any response ID.

### Response IDs this proxy did not issue

The proxy can only tell who owns a response when it issued that response's ID itself. An ID in any other shape, a raw provider ID or one handed out before response ID encryption was on, carries no owner, so the proxy refuses it with 403 on retrieve, cancel, delete, input items, and on `previous_response_id`:

```json
{
  "error": {
    "message": "Forbidden. This response id was not issued by this proxy, so the proxy cannot tell who owns it.",
    "code": 403
  }
}
```

Deployments that pass provider response IDs straight through on purpose, or that still have clients holding older IDs, turn the refusal off with `allow_unmanaged_response_ids`:

```yaml
general_settings:
  allow_unmanaged_response_ids: true
```

IDs the proxy did issue stay owner-checked either way, and proxy admin keys are exempt from both checks. `disable_responses_id_security: true` turns off the whole feature, this refusal included.

## Background Mode

LiteLLM passes OpenAI's `background: true` parameter through to the provider. The provider returns immediately with a response in `queued` or `in_progress` status, and you fetch the result later with `GET /v1/responses/{response_id}`:

```bash showLineNumbers title="Create a background response"
curl http://localhost:4000/v1/responses \
  -H "Authorization: Bearer $LITELLM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5.6",
    "input": "Write a detailed comparison of the Responses API and the Chat Completions API",
    "background": true
  }'
```

```bash showLineNumbers title="Poll for the result"
curl http://localhost:4000/v1/responses/{response_id} \
  -H "Authorization: Bearer $LITELLM_API_KEY"
```

### Cost tracking for background responses

The create call returns before the model has produced any tokens, so there is no usage to record at request time. To close that gap, the proxy stores every queued background response and prices it with a background polling job, the same machinery the [managed batches cost poller](./proxy/managed_batches#observability) uses for completed batches. It requires a Postgres database and ships with the enterprise package

Every `proxy_batch_polling_interval` seconds (a `general_settings` key, also settable via the `PROXY_BATCH_POLLING_INTERVAL` env var; default `3600`, plus up to 30s of jitter) the job reads pending background responses from the database, oldest first and up to `MAX_OBJECTS_PER_POLL_CYCLE` (default `50`) per cycle, and retrieves each one from the provider with the deployment credentials in your config. Once a response reaches a terminal status (`completed`, `failed`, `cancelled`, or `incomplete`), that retrieval writes a spend log with the final usage, attributed to the user who created the response, and the row stops being polled. Your own `GET /v1/responses/{response_id}` reads are never billed; only the poller's retrieval prices the response

Responses still pending after `MANAGED_OBJECT_STALENESS_CUTOFF_DAYS` (default `7`) days are marked stale and dropped from polling. Set the polling interval to something small like `30` while testing, and set `PROXY_BATCH_POLLING_ENABLED=false` to disable this job and the batch cost poller entirely

## Supported Responses API Parameters

| Provider | Supported Parameters |
|----------|---------------------|
| `openai` | [All Responses API parameters are supported](https://github.com/BerriAI/litellm/blob/7c3df984da8e4dff9201e4c5353fdc7a2b441831/litellm/llms/openai/responses/transformation.py#L23) |
| `azure` | [All Responses API parameters are supported](https://github.com/BerriAI/litellm/blob/7c3df984da8e4dff9201e4c5353fdc7a2b441831/litellm/llms/openai/responses/transformation.py#L23) |
| `azure_ai` on `.services.ai.azure.com` and `.openai.azure.com` hosts | [All Responses API parameters are supported](https://github.com/BerriAI/litellm/blob/913ef6ed49250f28680a1a50850183b5d80f6bbb/litellm/llms/azure_ai/responses/transformation.py#L25) |
| `azure_ai` on other hosts | [See supported parameters here](https://github.com/BerriAI/litellm/blob/f39d9178868662746f159d5ef642c7f34f9bfe5f/litellm/responses/litellm_completion_transformation/transformation.py#L57) |
| `anthropic` | [See supported parameters here](https://github.com/BerriAI/litellm/blob/f39d9178868662746f159d5ef642c7f34f9bfe5f/litellm/responses/litellm_completion_transformation/transformation.py#L57) |
| `bedrock` | [See supported parameters here](https://github.com/BerriAI/litellm/blob/f39d9178868662746f159d5ef642c7f34f9bfe5f/litellm/responses/litellm_completion_transformation/transformation.py#L57) |
| `gemini` | [See supported parameters here](https://github.com/BerriAI/litellm/blob/f39d9178868662746f159d5ef642c7f34f9bfe5f/litellm/responses/litellm_completion_transformation/transformation.py#L57) |
| `vertex_ai` | [See supported parameters here](https://github.com/BerriAI/litellm/blob/f39d9178868662746f159d5ef642c7f34f9bfe5f/litellm/responses/litellm_completion_transformation/transformation.py#L57) |
| All other llm api providers | [See supported parameters here](https://github.com/BerriAI/litellm/blob/f39d9178868662746f159d5ef642c7f34f9bfe5f/litellm/responses/litellm_completion_transformation/transformation.py#L57) |

## Load Balancing with Session Continuity.

When using the Responses API with multiple deployments of the same model (e.g., multiple Azure OpenAI endpoints), LiteLLM provides session continuity. This ensures that follow-up requests using a `previous_response_id` are routed to the same deployment that generated the original response.

#### Example Usage

**Python SDK**

```python showLineNumbers title="Python SDK with Session Continuity"
import litellm

# Set up router with multiple deployments of the same model
router = litellm.Router(
    model_list=[
        {
            "model_name": "azure-gpt4-turbo",
            "litellm_params": {
                "model": "azure/gpt-5.6-terra",
                "api_key": "your-api-key-1",
                "api_version": "2024-06-01",
                "api_base": "https://endpoint1.openai.azure.com",
            },
        },
        {
            "model_name": "azure-gpt4-turbo",
            "litellm_params": {
                "model": "azure/gpt-5.6-terra",
                "api_key": "your-api-key-2",
                "api_version": "2024-06-01",
                "api_base": "https://endpoint2.openai.azure.com",
            },
        },
    ],
    # `responses_api_deployment_check` ensures Requests with `previous_response_id`
    # are routed to the same deployment. `deployment_affinity` adds sticky sessions
    # for requests without `previous_response_id` (useful for implicit caching).
    # `session_affinity` adds sticky sessions based on `session_id` metadata.
    optional_pre_call_checks=["responses_api_deployment_check", "deployment_affinity", "session_affinity"],
    # Optional (default is 3600 seconds / 1 hour)
    deployment_affinity_ttl_seconds=3600,
)

# Initial request
response = await router.aresponses(
    model="azure-gpt4-turbo",
    input="Hello, who are you?",
    truncation="auto",
)

# Store the response ID
response_id = response.id

# Follow-up request - will be automatically routed to the same deployment
follow_up = await router.aresponses(
    model="azure-gpt4-turbo",
    input="Tell me more about yourself",
    truncation="auto",
    previous_response_id=response_id  # This ensures routing to the same deployment
)
```

**Proxy Server**

#### 1. Setup session continuity on proxy config.yaml

To enable session continuity for Responses API in your LiteLLM proxy, set `optional_pre_call_checks` in your proxy config.yaml.

- `responses_api_deployment_check`: high priority routing when `previous_response_id` is provided
- `encrypted_content_affinity`: **[Recommended]** content-aware routing for encrypted items (e.g., `rs_...` reasoning items) (**requires LiteLLM >= 1.82.3**)
- `session_affinity`: sticky sessions based on session id (takes priority over `deployment_affinity`)
- `deployment_affinity`: sticky sessions based on user key (applies even without `previous_response_id`)

:::tip[Recommended: Use `encrypted_content_affinity`]
For Responses API with load balancing across deployments with **different API keys**, use `encrypted_content_affinity` instead of `deployment_affinity`. It only pins requests that contain encrypted content, avoiding quota reduction while preventing `invalid_encrypted_content` errors. (Requires LiteLLM >= 1.82.3.)
:::

Notes:
- User-key affinity is keyed on `metadata.user_api_key_hash` (the API key hash). The OpenAI `user` request parameter is an end-user identifier and is intentionally not used for deployment affinity.
- Session-ID affinity is keyed on `metadata.session_id`. For proxy requests, this can be passed via the `x-litellm-session-id` or `x-litellm-trace-id` HTTP header (they are interchangeable for call chaining). For Python SDK requests, you can pass it via `litellm_metadata={"session_id": "value"}` in request args.
- `user_api_key_hash` is already SHA-256, and is used as-is (no double hashing).
- Affinity is scoped by a stable model identifier (the model-map key, e.g. `model_map_information.model_map_key`) so model aliases map to the same stickiness bucket.
- The mapping TTL is controlled by `deployment_affinity_ttl_seconds` (configured on Router init / proxy startup).

```yaml showLineNumbers title="config.yaml with Session Continuity"
model_list:
  - model_name: azure-gpt4-turbo
    litellm_params:
      model: azure/gpt-5.6-terra
      api_key: your-api-key-1
      api_version: 2024-06-01
      api_base: https://endpoint1.openai.azure.com
  - model_name: azure-gpt4-turbo
    litellm_params:
      model: azure/gpt-5.6-terra
      api_key: your-api-key-2
      api_version: 2024-06-01
      api_base: https://endpoint2.openai.azure.com

router_settings:
  optional_pre_call_checks:
    - responses_api_deployment_check
    - session_affinity
    - deployment_affinity
  # Optional (default is 3600 seconds / 1 hour)
  deployment_affinity_ttl_seconds: 3600
```

#### 2. Use the OpenAI Python SDK to make requests to LiteLLM Proxy

```python showLineNumbers title="OpenAI Client with Proxy Server"
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:4000",
    api_key="your-api-key"
)

# Initial request
response = client.responses.create(
    model="azure-gpt4-turbo",
    input="Hello, who are you?"
)

response_id = response.id

# Follow-up request - will be automatically routed to the same deployment
follow_up = client.responses.create(
    model="azure-gpt4-turbo",
    input="Tell me more about yourself",
    previous_response_id=response_id  # This ensures routing to the same deployment
)
```

## Encrypted Content Affinity (Multi-Region Load Balancing)

When load balancing Responses API across deployments with **different API keys** (e.g., different Azure regions or OpenAI organizations), encrypted content items (like `rs_...` reasoning items) can only be decrypted by the API key that created them.

### The Problem

```json
{
  "error": {
    "message": "The encrypted content for item rs_0d09d6e56879e76500699d6feee41c8197bd268aae76141f87 could not be verified. Reason: Encrypted content organization_id did not match the target organization.",
    "type": "invalid_request_error",
    "code": "invalid_encrypted_content"
  }
}
```

This error occurs when:
1. Initial request goes to Deployment A (API Key 1) → produces encrypted item `rs_xyz`
2. Follow-up request with `rs_xyz` in input gets load balanced to Deployment B (API Key 2)
3. Deployment B cannot decrypt content created by Deployment A → **request fails**

### The Solution: `encrypted_content_affinity`

The `encrypted_content_affinity` pre-call check routes follow-up requests containing encrypted items to the originating deployment **only when necessary**

**Key Benefits:**
- ✅ **No quota reduction**: Unlike `deployment_affinity`, only pins requests that contain encrypted items
- ✅ **Bypasses rate limits**: When encrypted content requires a specific deployment, RPM/TPM limits are bypassed (the request would fail on any other deployment anyway)
- ✅ **No `previous_response_id` required**: Works by encoding `model_id` directly into item IDs
- ✅ **No cache required**: `model_id` is decoded on-the-fly, so there is no Redis dependency and no TTL to manage
- ✅ **Globally safe**: Can be enabled for all models; non-Responses-API calls (chat, embeddings) are unaffected

### How It Works

1. **Encoding Phase** (on response):
   - For each output item that contains `encrypted_content`, LiteLLM rewrites the item ID to embed the originating `model_id`: `rs_xyz` → `encitem_{base64("litellm:model_id:{model_id};item_id:rs_xyz")}`
   - The original item ID is restored before forwarding the request to the upstream provider

2. **Routing Phase** (before request):
   - Scans request `input` for `encitem_` prefixed IDs
   - If found → decodes `model_id`, pins to originating deployment, bypasses rate limits
   - If no encoded items → normal load balancing

### When the originating deployment cannot serve the turn

The pin holds only while the originating deployment is in the healthy pool of the routed model group. When it is not, because it is cooled down after errors, it was removed from the config, or the follow-up was routed to a different model group (an auto-router tier change, or a client switching `model` between turns), LiteLLM first looks for a peer, a deployment whose resolved `api_base` and `api_key` are identical to the origin's, and pins to that instead. Deployments in different regions or with different keys never count as peers, whatever the provider would accept, so a multi-region group has none. An origin that was removed from the config, or an id that matches no deployment, has no credentials left to match and skips the peer search

Without a peer the turn is served in degraded form rather than failed. On the Responses API each reasoning item keeps its summary text and loses only its encrypted payload and id (an item with no readable text is dropped whole), and on a `/v1/messages` follow-up the thinking block is dropped whole. The rest of the conversation is untouched, the request goes to the healthy deployments through the normal routing strategy, and the model reasons fresh on that turn. The reasoning items it returns carry the id of the deployment that served it, so later turns pin there. Every degraded turn logs one router warning, so watch for it when reasoning continuity across turns matters to you:

```
EncryptedContentAffinityCheck: model_id=<id> cannot serve group <model> and no deployment on the same encryption boundary is configured; forwarding without its encrypted reasoning
```

Only a peer keeps the reasoning across such a turn. On an auto-router, `complexity_router_config.session_affinity: true` keeps a session that carries a `session_id` on the tier that produced the items (see [auto routing](./proxy/auto_routing.md)), so the turn usually stays with its origin, though escalation and routing plugins can still move it. Releases through v1.103.x failed a cooled-down origin with no peer with a 429 or 503 instead of serving the turn, and releases before v1.102.0 failed a removed origin or a group change the same way

The check can be turned on and off on a running proxy through `POST /config/update`, see [changing affinity settings at runtime](./routing.md#settings)

### Configuration

**Python SDK**

```python
from litellm import Router

router = Router(
    model_list=[
        {
            "model_name": "gpt-5.6-terra",
            "litellm_params": {
                "model": "openai/gpt-5.6-terra",
                "api_key": "org-1-api-key",  # Different API key
            },
            "model_info": {"id": "deployment-us-east"},
        },
        {
            "model_name": "gpt-5.6-terra",
            "litellm_params": {
                "model": "openai/gpt-5.6-terra",
                "api_key": "org-2-api-key",  # Different API key
            },
            "model_info": {"id": "deployment-eu-west"},
        },
    ],
    optional_pre_call_checks=["encrypted_content_affinity"],
)

# Initial request - routes to any deployment
response1 = await router.aresponses(
    model="gpt-5.6-terra",
    input="Explain quantum computing",
)

# Follow-up with encrypted items - automatically routes to same deployment
response2 = await router.aresponses(
    model="gpt-5.6-terra",
    input=response1.output,  # Contains encrypted items from response1
)
```

**Proxy Server**

```yaml showLineNumbers title="config.yaml"
model_list:
  - model_name: gpt-5.6-terra
    litellm_params:
      model: azure/gpt-5.6-terra
      api_base: https://eastus.openai.azure.com/
      api_key: os.environ/AZURE_API_KEY_EASTUS
      rpm: 600
      tpm: 100000
    model_info:
      id: "gpt-5.6-terra-eastus"

  - model_name: gpt-5.6-terra
    litellm_params:
      model: azure/gpt-5.6-terra
      api_base: https://westeurope.openai.azure.com/
      api_key: os.environ/AZURE_API_KEY_WESTEUROPE
      rpm: 600
      tpm: 100000
    model_info:
      id: "gpt-5.6-terra-westeurope"

router_settings:
  routing_strategy: usage-based-routing-v2
  enable_pre_call_checks: true
  optional_pre_call_checks:
    - encrypted_content_affinity
```

**Start proxy:**
```bash
litellm --config config.yaml
```

### When to Use Each Affinity Type

| Affinity Type | Use Case | Scope | Quota Impact |
|---------------|----------|-------|--------------|
| **`encrypted_content_affinity`** | **[Recommended]** Multi-region Responses API with different API keys | Only requests with tracked encrypted items | ✅ None (surgical pinning) |
| `responses_api_deployment_check` | When `previous_response_id` is available | Requests with `previous_response_id` | ✅ None |
| `session_affinity` | Session-based applications | All requests with same `session_id` | ⚠️ Reduces quota by # of sessions |
| `deployment_affinity` | Simple sticky sessions | All requests from same API key | ❌ Reduces quota by # of users |

## Per-Model-Group Affinity Configuration

By default, `optional_pre_call_checks` applies globally to all model groups. Use `model_group_affinity_config` when you want different affinity behavior per model group, for example enabling stickiness only for models spread across providers (Azure + Bedrock) while leaving single-provider groups free to load-balance.

Groups not listed fall back to the global `optional_pre_call_checks` settings.

**Python SDK**

```python
router = litellm.Router(
    model_list=[
        {
            "model_name": "gpt-5.6-terra",
            "litellm_params": {"model": "azure/gpt-5.6-terra", "api_key": "...", "api_base": "https://endpoint1.openai.azure.com"},
        },
        {
            "model_name": "gpt-5.6-terra",
            "litellm_params": {"model": "bedrock/us.anthropic.claude-sonnet-5", "aws_region_name": "us-east-1"},
        },
        {
            "model_name": "text-embedding-ada-002",
            "litellm_params": {"model": "azure/text-embedding-ada-002", "api_key": "...", "api_base": "https://endpoint1.openai.azure.com"},
        },
        {
            "model_name": "text-embedding-ada-002",
            "litellm_params": {"model": "azure/text-embedding-ada-002", "api_key": "...", "api_base": "https://endpoint2.openai.azure.com"},
        },
    ],
    # gpt-5.6-terra: cross-provider (Azure + Bedrock) — enable deployment affinity
    # text-embedding-ada-002: same provider — no affinity, let it load balance freely
    model_group_affinity_config={
        "gpt-5.6-terra": ["deployment_affinity", "responses_api_deployment_check"],
    },
)
```

**Proxy Server**

```yaml title="config.yaml"
model_list:
  - model_name: gpt-5.6-terra
    litellm_params:
      model: azure/gpt-5.6-terra
      api_key: os.environ/AZURE_API_KEY_1
      api_base: https://endpoint1.openai.azure.com

  - model_name: gpt-5.6-terra
    litellm_params:
      model: bedrock/us.anthropic.claude-sonnet-5
      aws_region_name: us-east-1

  - model_name: text-embedding-ada-002
    litellm_params:
      model: azure/text-embedding-ada-002
      api_key: os.environ/AZURE_API_KEY_1
      api_base: https://endpoint1.openai.azure.com

  - model_name: text-embedding-ada-002
    litellm_params:
      model: azure/text-embedding-ada-002
      api_key: os.environ/AZURE_API_KEY_2
      api_base: https://endpoint2.openai.azure.com

router_settings:
  # gpt-5.6-terra: cross-provider — enable stickiness
  # text-embedding-ada-002: not listed — load balances freely
  model_group_affinity_config:
    "gpt-5.6-terra":
      - deployment_affinity
      - responses_api_deployment_check
```

**Supported values:** `deployment_affinity`, `responses_api_deployment_check`, `session_affinity`

## Calling non-Responses API endpoints (`/responses` to `/chat/completions` Bridge)

LiteLLM allows you to call non-Responses API models via a bridge to LiteLLM's `/chat/completions` endpoint. This is useful for calling Anthropic, Gemini and even non-Responses API OpenAI models.

#### Python SDK Usage

```python showLineNumbers title="SDK Usage"
import litellm
import os

# Set API key
os.environ["ANTHROPIC_API_KEY"] = "your-anthropic-api-key"

# Non-streaming response
response = litellm.responses(
    model="anthropic/claude-sonnet-5",
    input="Tell me a three sentence bedtime story about a unicorn.",
    max_output_tokens=100
)

print(response)
```

#### LiteLLM Proxy Usage

**Setup Config:**

```yaml showLineNumbers title="Example Configuration"
model_list:
- model_name: anthropic-model
  litellm_params:
    model: anthropic/claude-sonnet-5
    api_key: os.environ/ANTHROPIC_API_KEY
```

**Start Proxy:**

```bash showLineNumbers title="Start LiteLLM Proxy"
litellm --config /path/to/config.yaml

# RUNNING on http://0.0.0.0:4000
```

**Make Request:**

```bash showLineNumbers title="non-Responses API Model Request"
curl http://localhost:4000/v1/responses \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $LITELLM_API_KEY" \
  -d '{
    "model": "anthropic-model",
    "input": "who is Michael Jordan"
  }'
```

### Opt-in bridge for `openai/` models with custom `api_base`

If you're using an **OpenAI-compatible third-party provider** (e.g. llama.cpp, vLLM, LM Studio) via `openai/` prefix with a custom `api_base`, LiteLLM will normally forward `/responses` requests directly to that endpoint. If the provider only supports `/chat/completions`, the request will fail.

Use either of these to force the `/responses` → `/chat/completions` bridge:

1. **`use_chat_completions_api: true`** — makes it explicit that LiteLLM will call the provider’s chat-completions API.
2. **`openai/chat_completions/<model_name>`** — same pattern as `responses/` on chat completions: the model id encodes the routing choice.

#### Python SDK Usage

```python showLineNumbers title="Force bridge for custom openai/ endpoint (flag)"
import litellm

response = litellm.responses(
    model="openai/my-custom-model",
    input="Hello!",
    api_base="http://localhost:8080",
    api_key="fake-key",
    use_chat_completions_api=True,
)

print(response)
```

Or encode it in the model id:

```python showLineNumbers title="Force bridge via openai/chat_completions/ model prefix"
import litellm

response = litellm.responses(
    model="openai/chat_completions/my-custom-model",
    input="Hello!",
    api_base="http://localhost:8080",
    api_key="fake-key",
)

print(response)
```

#### LiteLLM Proxy Usage

**Setup Config:**

```yaml showLineNumbers title="config.yaml — bridge for custom openai/ endpoint"
model_list:
- model_name: my-local-model
  litellm_params:
    model: openai/my-custom-model
    api_base: http://localhost:8080/v1
    api_key: fake-key
    use_chat_completions_api: true
```

Alternatively set `model: openai/chat_completions/my-custom-model` instead of the flag.

**Start Proxy:**

```bash showLineNumbers title="Start LiteLLM Proxy"
litellm --config /path/to/config.yaml

# RUNNING on http://0.0.0.0:4000
```

**Make Request:**

```bash showLineNumbers title="Request via bridge"
curl http://localhost:4000/v1/responses \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $LITELLM_API_KEY" \
  -d '{
    "model": "my-local-model",
    "input": "Hello!"
  }'
```

This is particularly useful when connecting clients that hardcode the `/responses` endpoint (e.g. OpenAI Codex CLI with `wire_api = "responses"`) to local or third-party OpenAI-compatible providers that only expose `/chat/completions`.

## Server-side compaction

For long-running conversations, you can enable **server-side compaction** so that when the rendered context size crosses a threshold, the server automatically runs compaction in-stream and emits a compaction item. No separate `POST /v1/responses/compact` call is required.

Supported on the OpenAI Responses API when using the `openai` or `azure` provider. Pass `context_management` with a compaction entry and `compact_threshold` (token count; minimum 1000). When the context crosses the threshold, the server compacts in-stream and continues. Chain turns with `previous_response_id` or by appending output items to your next input array. See [OpenAI Compaction guide](https://developers.openai.com/api/docs/guides/compaction) for details.

> **Note:** You can use openai `context_management` format with Anthropic models via LiteLLM via responses API. LiteLLM will automatically translate this format for Anthropic and handle context management for you.

For explicit control over when compaction runs, use the standalone compact endpoint (`POST /v1/responses/compact`) instead.

### Python SDK

```python showLineNumbers title="Server-side compaction with LiteLLM Python SDK"
import litellm

# Non-streaming: enable compaction when context exceeds 200k tokens
response = litellm.responses(
    model="openai/gpt-5.6-terra",
    input="Your conversation input...",
    context_management=[{"type": "compaction", "compact_threshold": 200000}],
    max_output_tokens=1024,
)
print(response)

# Streaming: same context_management, compaction runs in-stream if threshold is crossed
stream = litellm.responses(
    model="openai/gpt-5.6-terra",
    input="Your conversation input...",
    context_management=[{"type": "compaction", "compact_threshold": 200000}],
    stream=True,
)
for event in stream:
    print(event)
```

### LiteLLM Proxy (AI Gateway)

Use the OpenAI SDK with your proxy as `base_url`, or call the proxy with curl. The proxy forwards `context_management` to the provider.

**OpenAI Python SDK (proxy as base_url):**

```python showLineNumbers title="Server-side compaction via LiteLLM Proxy"
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:4000",  # LiteLLM Proxy (AI Gateway)
    api_key="your-proxy-api-key",
)

response = client.responses.create(
    model="openai/gpt-5.6-terra",
    input="Your conversation input...",
    context_management=[{"type": "compaction", "compact_threshold": 200000}],
    max_output_tokens=1024,
)
print(response)
```

**curl (proxy):**

```bash title="Server-side compaction via curl to LiteLLM Proxy"
curl -X POST "http://localhost:4000/v1/responses" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-proxy-api-key" \
  -d '{
    "model": "openai/gpt-5.6-terra",
    "input": "Your conversation input...",
    "context_management": [{"type": "compaction", "compact_threshold": 200000}],
    "max_output_tokens": 1024
  }'
```

## Shell tool

The **Shell tool** lets the model run commands in a hosted container or local runtime (OpenAI Responses API). You pass `tools=[{"type": "shell", "environment": {...}}]`; the `environment` object configures the runtime (e.g. `type: "container_auto"` for auto-provisioned containers). See [OpenAI Shell tool guide](https://developers.openai.com/api/docs/guides/tools-shell) for full options.

Supported when using the `openai` or `azure` provider with a model that supports the Shell tool.

### Python SDK

```python showLineNumbers title="Shell tool with LiteLLM Python SDK"
import litellm

response = litellm.responses(
    model="openai/gpt-5.6-terra",
    input="List files in /mnt/data and run python --version.",
    tools=[{"type": "shell", "environment": {"type": "container_auto"}}],
    tool_choice="auto",
    max_output_tokens=1024,
)
```

### LiteLLM Proxy (AI Gateway)

Use the OpenAI SDK with your proxy as `base_url`, or call the proxy with curl. The proxy forwards `tools` (including `type: "shell"`) to the provider.

**OpenAI Python SDK (proxy as base_url):**

```python showLineNumbers title="Shell tool via LiteLLM Proxy"
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:4000",
    api_key="your-proxy-api-key",
)

response = client.responses.create(
    model="openai/gpt-5.6-terra",
    input="List files in /mnt/data.",
    tools=[{"type": "shell", "environment": {"type": "container_auto"}}],
    tool_choice="auto",
    max_output_tokens=1024,
)
```

**curl:**

```bash title="Shell tool via curl to LiteLLM Proxy"
curl -X POST "http://localhost:4000/v1/responses" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-proxy-api-key" \
  -d '{
    "model": "openai/gpt-5.6-terra",
    "input": "List files in /mnt/data.",
    "tools": [{"type": "shell", "environment": {"type": "container_auto"}}],
    "tool_choice": "auto",
    "max_output_tokens": 1024
  }'
```

## File Search (Vector Stores)

For full `file_search` usage (native + emulated fallback), SDK/Proxy examples, architecture diagram, and Q&A, see:

- [`File Search in the Responses API: E2E Testing Guide`](/docs/tutorials/file_search_responses_api)

## Session Management

LiteLLM Proxy supports session management for all supported models. This allows you to store and fetch conversation history (state) in LiteLLM Proxy. 

#### Usage

1. Enable storing request / response content in the database

Set `store_prompts_in_spend_logs: true` under `general_settings` and `cold_storage_custom_logger: s3_v2` under `litellm_settings` in your proxy config.yaml. When this is enabled, LiteLLM will store the request and response content in the s3 bucket you specify.

```yaml showLineNumbers title="config.yaml with Session Continuity"
litellm_settings:
  callbacks: ["s3_v2"]
  cold_storage_custom_logger: s3_v2
  s3_callback_params: # learn more https://docs.litellm.ai/docs/proxy/logging#s3-buckets
    s3_bucket_name: litellm-logs   # AWS Bucket Name for S3
    s3_region_name: us-west-2      

general_settings:
  store_prompts_in_spend_logs: true
```

2. Make request 1 with no `previous_response_id` (new session)

Start a new conversation by making a request without specifying a previous response ID.

**Curl**

```curl
curl http://localhost:4000/v1/responses \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $LITELLM_API_KEY" \
  -d '{
    "model": "anthropic/claude-sonnet-5",
    "input": "who is Michael Jordan"
  }'
```

**OpenAI Python SDK**

```python
from openai import OpenAI

# Initialize the client with your LiteLLM proxy URL
client = OpenAI(
    base_url="http://localhost:4000",
    api_key="sk-<your-litellm-api-key>"
)

# Make initial request to start a new conversation
response = client.responses.create(
    model="anthropic/claude-sonnet-5",
    input="who is Michael Jordan"
)

print(response.id)  # Store this ID for future requests in same session
print(response.output[0].content[0].text)
```

Response:

```json
{
  "id":"resp_123abc",
  "model":"claude-sonnet-5",
  "output":[{
    "type":"message",
    "content":[{
      "type":"output_text",
      "text":"Michael Jordan is widely considered one of the greatest basketball players of all time. He played for the Chicago Bulls (1984-1993, 1995-1998) and Washington Wizards (2001-2003), winning 6 NBA Championships with the Bulls."
    }]
  }]
}
```

3. Make request 2 with `previous_response_id` (same session)

Continue the conversation by referencing the previous response ID to maintain conversation context.

**Curl**

```curl
curl http://localhost:4000/v1/responses \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $LITELLM_API_KEY" \
  -d '{
    "model": "anthropic/claude-sonnet-5",
    "input": "can you tell me more about him",
    "previous_response_id": "resp_123abc"
  }'
```

**OpenAI Python SDK**

```python
from openai import OpenAI

# Initialize the client with your LiteLLM proxy URL
client = OpenAI(
    base_url="http://localhost:4000",
    api_key="sk-<your-litellm-api-key>"
)

# Make follow-up request in the same conversation session
follow_up_response = client.responses.create(
    model="anthropic/claude-sonnet-5",
    input="can you tell me more about him",
    previous_response_id="resp_123abc"  # ID from the previous response
)

print(follow_up_response.output[0].content[0].text)
```

Response:

```json
{
  "id":"resp_456def",
  "model":"claude-sonnet-5",
  "output":[{
    "type":"message",
    "content":[{
      "type":"output_text",
      "text":"Michael Jordan was born February 17, 1963. He attended University of North Carolina before being drafted 3rd overall by the Bulls in 1984. Beyond basketball, he built the Air Jordan brand with Nike and later became owner of the Charlotte Hornets."
    }]
  }]
}
```

4. Make request 3 with no `previous_response_id` (new session)

Start a brand new conversation without referencing previous context to demonstrate how context is not maintained between sessions.

**Curl**

```curl
curl http://localhost:4000/v1/responses \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $LITELLM_API_KEY" \
  -d '{
    "model": "anthropic/claude-sonnet-5",
    "input": "can you tell me more about him"
  }'
```

**OpenAI Python SDK**

```python
from openai import OpenAI

# Initialize the client with your LiteLLM proxy URL
client = OpenAI(
    base_url="http://localhost:4000",
    api_key="sk-<your-litellm-api-key>"
)

# Make a new request without previous context
new_session_response = client.responses.create(
    model="anthropic/claude-sonnet-5",
    input="can you tell me more about him"
    # No previous_response_id means this starts a new conversation
)

print(new_session_response.output[0].content[0].text)
```

Response:

```json
{
  "id":"resp_789ghi",
  "model":"claude-sonnet-5",
  "output":[{
    "type":"message",
    "content":[{
      "type":"output_text",
      "text":"I don't see who you're referring to in our conversation. Could you let me know which person you'd like to learn more about?"
    }]
  }]
}
```

## Related pages

- [embedding()](https://docs.litellm.ai/en/docs/embedding/supported_embedding.md)
- [text_completion()](https://docs.litellm.ai/en/docs/text_completion.md)