---
title: "VLLM"
url: "/docs/providers/vllm"
canonical_url: "https://docs.litellm.ai/docs/providers/vllm"
type: "docs"
last_updated: "2026-10-08"
related:
  - "/docs/providers/vercel_ai_gateway"
  - "/docs/providers/vllm_batches"
---
# VLLM

> Index of all LiteLLM docs: https://docs.litellm.ai/llms.txt


LiteLLM 支援 VLLM 上的所有模型。

| 屬性 | 詳細資訊 |
|-------|-------|
| 說明 | vLLM 是一個快速且易於使用的 LLM 推論與服務程式庫。[文件](https://docs.vllm.ai/en/latest/index.html) |
| LiteLLM 提供者路由 | `hosted_vllm/`（用於 OpenAI 相容伺服器），`vllm/`（[已棄用]，用於 vLLM sdk 使用） |
| 提供者文件 | [vLLM ↗](https://docs.vllm.ai/en/latest/index.html) |
| 支援的端點 | `/chat/completions`、`/embeddings`、`/completions`、`/rerank`、`/audio/transcriptions` |

## 快速開始 {#quick-start}

## 使用 - litellm.completion（呼叫 OpenAI 相容端點） {#usage---litellmcompletion-calling-openai-compatible-endpoint}
vLLM 提供 OpenAI 相容端點 - 以下是如何透過 LiteLLM 呼叫它

若要使用 litellm 呼叫代管的 vllm 伺服器，請將下列內容加入您的 completion 呼叫中

* `model="hosted_vllm/<your-vllm-model-name>"` 
* `api_base = "your-hosted-vllm-server"`

```python
import litellm 

response = litellm.completion(
            model="hosted_vllm/facebook/opt-125m", # pass the vllm model name
            messages=messages,
            api_base="https://hosted-vllm-api.co",
            temperature=0.2,
            max_tokens=80)

print(response)
```

## 使用 -  LiteLLM Proxy Server（呼叫 OpenAI 相容端點） {#usage----litellm-proxy-server-calling-openai-compatible-endpoint}

以下是如何使用 LiteLLM Proxy Server 呼叫 OpenAI 相容端點

1. 修改 config.yaml 

  ```yaml
  model_list:
    - model_name: my-model
      litellm_params:
        model: hosted_vllm/facebook/opt-125m  # add hosted_vllm/ prefix to route as OpenAI provider
        api_base: https://hosted-vllm-api.co      # add api base for OpenAI compatible provider
  ```

2. 啟動 proxy 

  ```bash
  $ litellm --config /path/to/config.yaml
  ```

3. 將請求送至 LiteLLM Proxy Server

**OpenAI Python v1.0.0+**

  ```python
  import openai
  client = openai.OpenAI(
      api_key="sk-<your-litellm-api-key>",             # pass litellm proxy key, if you're using virtual keys
      base_url="http://0.0.0.0:4000" # litellm-proxy-base url
  )

  response = client.chat.completions.create(
      model="my-model",
      messages = [
          {
              "role": "user",
              "content": "what llm are you"
          }
      ],
  )

  print(response)
  ```

**curl**

  ```shell
  curl --location 'http://0.0.0.0:4000/chat/completions' \
      --header "Authorization: Bearer $LITELLM_API_KEY" \
      --header 'Content-Type: application/json' \
      --data '{
      "model": "my-model",
      "messages": [
          {
          "role": "user",
          "content": "what llm are you"
          }
      ],
  }'
  ```

  ## Reasoning Effort

**SDK**

  ```python
  from litellm import completion

  response = completion(
      model="hosted_vllm/gpt-oss-120b",
      messages=[{"role": "user", "content": "whats 2 + 2"}],
      reasoning_effort="high",
      api_base="https://hosted-vllm-api.co",
  )
  print(response)
  ```
**PROXY**

  1. 設定 config.yaml

  ```yaml
  model_list:
    - model_name: gpt-oss-120b
      litellm_params:
        model: hosted_vllm/gpt-oss-120b
        api_base: https://hosted-vllm-api.co
  ```

  2. 啟動 proxy

  ```bash
  litellm --config /path/to/config.yaml
  ```

  3. 測試它！

  ```bash
  curl http://0.0.0.0:4000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{"model": "gpt-oss-120b", "messages": [{"role": "user", "content": "whats 2 + 2"}], "reasoning_effort": "high"}'
  ```

## 嵌入 {#embeddings}

vLLM 提供 OpenAI 相容的 `/v1/embeddings`。當用戶端省略 `encoding_format` 時，LiteLLM 會為 OpenAI 相容的 embedding 路由預設設定它（request → model `litellm_params` → `LITELLM_DEFAULT_EMBEDDING_ENCODING_FORMAT` → `float`）。請參閱 [Embeddings](../proxy/embedding.md#embedding-encoding-format)。

**SDK**

```python
from litellm import embedding   
import os

os.environ["HOSTED_VLLM_API_BASE"] = "http://localhost:8000"

embedding = embedding(model="hosted_vllm/facebook/opt-125m", input=["Hello world"])

print(embedding)
```

**PROXY**

1. 設定 config.yaml

```yaml
model_list:
    - model_name: my-model
      litellm_params:
        model: hosted_vllm/facebook/opt-125m  # add hosted_vllm/ prefix to route as OpenAI provider
        api_base: https://hosted-vllm-api.co      # add api base for OpenAI compatible provider
```

2. 啟動 proxy 

```bash
$ litellm --config /path/to/config.yaml

# RUNNING on http://0.0.0.0:4000
```

3. 測試它！ 

```bash
curl -L -X POST 'http://0.0.0.0:4000/embeddings' \
-H "Authorization: Bearer $LITELLM_API_KEY" \
-H 'Content-Type: application/json' \
-d '{"input": ["hello world"], "model": "my-model"}'
```

[查看 OpenAI SDK/Langchain/etc. 範例](../proxy/user_keys.md#embeddings)

## 重排 {#rerank}

**SDK**

```python
from litellm import rerank
import os

os.environ["HOSTED_VLLM_API_BASE"] = "http://localhost:8000"
os.environ["HOSTED_VLLM_API_KEY"] = ""  # [optional], if your VLLM server requires an API key

query = "What is the capital of the United States?"
documents = [
    "Carson City is the capital city of the American state of Nevada.",
    "The Commonwealth of the Northern Mariana Islands is a group of islands in the Pacific Ocean. Its capital is Saipan.",
    "Washington, D.C. is the capital of the United States.",
    "Capital punishment has existed in the United States since before it was a country.",
]

response = rerank(
    model="hosted_vllm/your-rerank-model",
    query=query,
    documents=documents,
    top_n=3,
)
print(response)
```

### 非同步使用 {#async-usage}

```python
from litellm import arerank
import os, asyncio

os.environ["HOSTED_VLLM_API_BASE"] = "http://localhost:8000"
os.environ["HOSTED_VLLM_API_KEY"] = ""  # [optional], if your VLLM server requires an API key

async def test_async_rerank(): 
    query = "What is the capital of the United States?"
    documents = [
        "Carson City is the capital city of the American state of Nevada.",
        "The Commonwealth of the Northern Mariana Islands is a group of islands in the Pacific Ocean. Its capital is Saipan.",
        "Washington, D.C. is the capital of the United States.",
        "Capital punishment has existed in the United States since before it was a country.",
    ]

    response = await arerank(
        model="hosted_vllm/your-rerank-model",
        query=query,
        documents=documents,
        top_n=3,
    )
    print(response)

asyncio.run(test_async_rerank())
```

**PROXY**

1. 設定 config.yaml

```yaml
model_list:
    - model_name: my-rerank-model
      litellm_params:
        model: hosted_vllm/your-rerank-model  # add hosted_vllm/ prefix to route as VLLM provider
        api_base: http://localhost:8000      # add api base for your VLLM server
        # api_key: your-api-key             # [optional] if your VLLM server requires authentication
```

2. 啟動 proxy 

```bash
$ litellm --config /path/to/config.yaml

# RUNNING on http://0.0.0.0:4000
```

3. 測試它！ 

```bash
curl -L -X POST 'http://0.0.0.0:4000/rerank' \
-H "Authorization: Bearer $LITELLM_API_KEY" \
-H 'Content-Type: application/json' \
-d '{
    "model": "my-rerank-model",
    "query": "What is the capital of the United States?",
    "documents": [
        "Carson City is the capital city of the American state of Nevada.",
        "The Commonwealth of the Northern Mariana Islands is a group of islands in the Pacific Ocean. Its capital is Saipan.",
        "Washington, D.C. is the capital of the United States.",
        "Capital punishment has existed in the United States since before it was a country."
    ],
    "top_n": 3
}'
```

[查看 OpenAI SDK/Langchain/etc. 範例](../rerank.md#litellm-proxy-usage)

### 截斷長文件 {#truncating-long-documents}

vLLM 的 `/rerank` 端點具有自己的截斷控制，而 LiteLLM 在設定後會將其轉發至 `hosted_vllm` rerank 模型。若沒有其中之一，超過 reranker context 視窗的文件會因 context 長度錯誤而失敗

| 參數 | 型別 | 描述 |
|-----------|------|-------------|
| `truncate_prompt_tokens` | integer | 在評分前，將每組查詢與文件配對截斷為這麼多 tokens |
| `truncation_side` | `left` 或 `right` | 輸入內容要截斷的端點 |
| `max_tokens_per_doc` | integer | 將每份文件上限設為這麼多 tokens |
| `max_tokens_per_query` | integer | 將查詢上限設為這麼多 tokens |

無效值（例如 `truncation_side: "middle"`）會在送往 vLLM 之前先回傳 400

**SDK**

```python
from litellm import rerank

response = rerank(
    model="hosted_vllm/your-rerank-model",
    query="What is the capital of the United States?",
    documents=["<a document longer than the reranker context window>"],
    truncate_prompt_tokens=512,
    truncation_side="left",
)
print(response)
```

**PROXY**

```bash
curl -L -X POST 'http://0.0.0.0:4000/rerank' \
-H "Authorization: Bearer $LITELLM_API_KEY" \
-H 'Content-Type: application/json' \
-d '{
    "model": "my-rerank-model",
    "query": "What is the capital of the United States?",
    "documents": ["<a document longer than the reranker context window>"],
    "truncate_prompt_tokens": 512,
    "truncation_side": "left"
}'
```

## 圖片編輯 {#image-edits}

vLLM-Omni 針對 OpenAI 相容的 `/v1/images/edits` 提供影像編輯模型，例如 `Qwen/Qwen-Image-Edit-2511`。請使用 `hosted_vllm/` 前綴，並將 `api_base` 指向 vLLM-Omni 伺服器；其他提供者欄位，例如 `seed` 或 `negative_prompt`，會作為表單欄位傳遞。vLLM-Omni 沒有 `quality` 或 `input_fidelity` 表單欄位，並且接受其遮罩為 `mask_image`（URL 字串），而不是 OpenAI 的 `mask` 檔案，因此 LiteLLM 會拒絕 `mask`、`quality` 和 `input_fidelity` 用於 `hosted_vllm/` 模型，除非設定了 `drop_params: true`；在此情況下，它們會在請求送出前被捨棄。若要遮罩編輯，請將 `mask_image` 作為額外欄位，並提供遮罩的 URL。

**SDK**

```python
from litellm import image_edit
import os

os.environ["HOSTED_VLLM_API_BASE"] = "http://localhost:8091"

response = image_edit(
    model="hosted_vllm/Qwen/Qwen-Image-Edit-2511",
    image=open("original_image.png", "rb"),
    prompt="Add a red hat to the person in the image",
)

print(response)
```

**PROXY**

1. 設定 config.yaml

```yaml
model_list:
    - model_name: qwen-image-edit
      litellm_params:
        model: hosted_vllm/Qwen/Qwen-Image-Edit-2511  # add hosted_vllm/ prefix to route as OpenAI provider
        api_base: http://localhost:8091              # your vLLM-Omni server
      model_info:
        mode: image_edit
```

2. 啟動 proxy

```bash
$ litellm --config /path/to/config.yaml

# RUNNING on http://0.0.0.0:4000
```

3. 測試它！

```bash
curl -X POST 'http://0.0.0.0:4000/v1/images/edits' \
-H "Authorization: Bearer $LITELLM_API_KEY" \
-F 'model=qwen-image-edit' \
-F 'image=@original_image.png' \
-F 'prompt=Add a red hat to the person in the image'
```

## 將影片 URL 傳送至 VLLM {#send-video-url-to-vllm}

VLLM 的範例實作 [請見此處](https://github.com/vllm-project/vllm/pull/10020)

**（統一）Files Message**

使用此方式可用 OpenAI 的 `files` 訊息類型，以相同格式將影片 URL 傳送至 VLLM + Gemini。

將影片 URL 傳送至 VLLM 有兩種方式：

1. 直接傳遞影片 URL

```
{"type": "file", "file": {"file_id": video_url}},
```

2. 將影片資料以 base64 傳遞

```
{"type": "file", "file": {"file_data": f"data:video/mp4;base64,{video_data_base64}"}}
```

**SDK**

```python
from litellm import completion

messages=[
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "Summarize the following video"
            },
            {
                "type": "file",
                "file": {
                    "file_id": "https://www.youtube.com/watch?v=dQw4w9WgXcQ"
                }
            }
        ]
    }
]

# call vllm 
os.environ["HOSTED_VLLM_API_BASE"] = "https://hosted-vllm-api.co"
os.environ["HOSTED_VLLM_API_KEY"] = "" # [optional], if your VLLM server requires an API key
response = completion(
    model="hosted_vllm/qwen", # pass the vllm model name
    messages=messages,
)

# call gemini 
os.environ["GEMINI_API_KEY"] = "your-gemini-api-key"
response = completion(
    model="gemini/gemini-3.8-flash", # pass the gemini model name
    messages=messages,
)

print(response)
```

**PROXY**

1. 設定 config.yaml

```yaml
model_list:
    - model_name: my-model
      litellm_params:
        model: hosted_vllm/qwen  # add hosted_vllm/ prefix to route as OpenAI provider
        api_base: https://hosted-vllm-api.co      # add api base for OpenAI compatible provider
    - model_name: my-gemini-model
      litellm_params:
        model: gemini/gemini-3.8-flash  # add gemini/ prefix to route as Google AI Studio provider
        api_key: os.environ/GEMINI_API_KEY
```

2. 啟動 proxy 

```bash
$ litellm --config /path/to/config.yaml

# RUNNING on http://0.0.0.0:4000
```

3. 測試它！ 

```bash
curl -X POST http://0.0.0.0:4000/chat/completions \
-H "Authorization: Bearer $LITELLM_API_KEY" \
-H "Content-Type: application/json" \
-d '{
    "model": "my-model",
    "messages": [
        {"role": "user", "content": 
            [
                {"type": "text", "text": "Summarize the following video"},
                {"type": "file", "file": {"file_id": "https://www.youtube.com/watch?v=dQw4w9WgXcQ"}}
            ]
        }
    ]
}'
```

**（VLLM 專用）Video Message**

使用此方式可將影片 URL 以其原生訊息格式（`video_url`）傳送至 VLLM。

將影片 URL 傳送至 VLLM 有兩種方式：

1. 直接傳遞影片 URL

```
{"type": "video_url", "video_url": {"url": video_url}},
```

2. 將影片資料以 base64 傳遞

```
{"type": "video_url", "video_url": {"url": f"data:video/mp4;base64,{video_data_base64}"}}
```

**SDK**

```python
from litellm import completion

response = completion(
            model="hosted_vllm/qwen", # pass the vllm model name
            messages=[
                {
                    "role": "user",
                    "content": [
                        {
                            "type": "text",
                            "text": "Summarize the following video"
                        },
                        {
                            "type": "video_url",
                            "video_url": {
                                "url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ"
                            }
                        }
                    ]
                }
            ],
            api_base="https://hosted-vllm-api.co")

print(response)
```

**PROXY**

1. 設定 config.yaml

```yaml
model_list:
    - model_name: my-model
      litellm_params:
        model: hosted_vllm/qwen  # add hosted_vllm/ prefix to route as OpenAI provider
        api_base: https://hosted-vllm-api.co      # add api base for OpenAI compatible provider
```

2. 啟動 proxy 

```bash
$ litellm --config /path/to/config.yaml

# RUNNING on http://0.0.0.0:4000
```

3. 測試它！ 

```bash
curl -X POST http://0.0.0.0:4000/chat/completions \
-H "Authorization: Bearer $LITELLM_API_KEY" \
-H "Content-Type: application/json" \
-d '{
    "model": "my-model",
    "messages": [
        {"role": "user", "content": 
            [
                {"type": "text", "text": "Summarize the following video"},
                {"type": "video_url", "video_url": {"url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ"}}
            ]
        }
    ]
}'
```

## （已棄用）用於套件化 `vllm` 安裝 {#deprecated-for-packaged-vllm-installs}

### 使用 - `litellm.completion` {#using---litellmcompletion}

```
uv add litellm vllm
```
```python
import litellm 

response = litellm.completion(
            model="vllm/facebook/opt-125m", # add a vllm prefix so litellm knows the custom_llm_provider==vllm
            messages=messages,
            temperature=0.2,
            max_tokens=80)

print(response)
```

### 批次完成 {#batch-completion}

```python
from litellm import batch_completion

model_name = "facebook/opt-125m"
provider = "vllm"
messages = [[{"role": "user", "content": "Hey, how's it going"}] for _ in range(5)]

response_list = batch_completion(
            model=model_name, 
            custom_llm_provider=provider, # can easily switch to huggingface, replicate, together ai, sagemaker, etc.
            messages=messages,
            temperature=0.2,
            max_tokens=80,
        )
print(response_list)
```

### 提示詞樣板 {#prompt-templates}

對於具有特殊提示詞樣板的模型（例如 Llama2），我們會將提示詞格式化以符合其樣板。

**如果我們不支援您需要的模型怎麼辦？**
您也可以指定自己的自訂提示詞格式化方式，以防我們尚未涵蓋您的模型。 

**這是否表示您必須為所有模型指定提示詞？**
不用。預設情況下，我們會將您的訊息內容串接成提示詞（適用於 Bloom、T-5、Llama-2 base models 等預期格式）

**預設提示詞樣板**
```python
def default_pt(messages):
    return " ".join(message["content"] for message in messages)
```

[LiteLLM 中提示詞樣板運作方式的程式碼](https://github.com/BerriAI/litellm/blob/main/litellm/llms/prompt_templates/factory.py)

#### 我們已經有提示詞樣板的模型 {#models-we-already-have-prompt-templates-for}

| 模型名稱                           | Works for Models                  | 函式呼叫                                                                                                    |
|--------------------------------------|-----------------------------------|------------------------------------------------------------------------------------------------------------------|
| meta-llama/Llama-2-7b-chat           | All meta-llama llama2 chat models | `completion(model='vllm/meta-llama/Llama-2-7b', messages=messages, api_base="your_api_endpoint")`                |
| tiiuae/falcon-7b-instruct            | All falcon instruct models        | `completion(model='vllm/tiiuae/falcon-7b-instruct', messages=messages, api_base="your_api_endpoint")`            |
| mosaicml/mpt-7b-chat                 | All mpt chat models               | `completion(model='vllm/mosaicml/mpt-7b-chat', messages=messages, api_base="your_api_endpoint")`                 |
| codellama/CodeLlama-34b-Instruct-hf  | All codellama instruct models     | `completion(model='vllm/codellama/CodeLlama-34b-Instruct-hf', messages=messages, api_base="your_api_endpoint")`  |
| WizardLM/WizardCoder-Python-34B-V1.0 | All wizardcoder models            | `completion(model='vllm/WizardLM/WizardCoder-Python-34B-V1.0', messages=messages, api_base="your_api_endpoint")` |
| Phind/Phind-CodeLlama-34B-v2         | All phind-codellama models        | `completion(model='vllm/Phind/Phind-CodeLlama-34B-v2', messages=messages, api_base="your_api_endpoint")`         |

#### 自訂提示詞樣板 {#custom-prompt-templates}

```python 
# Create your own custom prompt template works 
litellm.register_prompt_template(
	model="togethercomputer/LLaMA-2-7B-32K",
	roles={
            "system": {
                "pre_message": "[INST] <<SYS>>\n",
                "post_message": "\n<</SYS>>\n [/INST]\n"
            },
            "user": { 
                "pre_message": "[INST] ",
                "post_message": " [/INST]\n"
            }, 
            "assistant": {
                "pre_message": "\n",
                "post_message": "\n",
            }
        } # tell LiteLLM how you want to map the openai messages to this model
)

def test_vllm_custom_model():
    model = "vllm/togethercomputer/LLaMA-2-7B-32K"
    response = completion(model=model, messages=messages)
    print(response['choices'][0]['message']['content'])
    return response

test_vllm_custom_model()
```

[實作程式碼](https://github.com/BerriAI/litellm/blob/6b3cb1898382f2e4e80fd372308ea232868c78d1/litellm/utils.py#L1414)

## Related pages

- [Vercel AI Gateway](https://docs.litellm.ai/docs/providers/vercel_ai_gateway.md)
- [vLLM - 批次 + 檔案 API](https://docs.litellm.ai/docs/providers/vllm_batches.md)