跳至主要內容

VLLM

LiteLLM 支援 VLLM 上的所有模型。

屬性詳細資訊
說明vLLM 是一個快速且易於使用的 LLM 推論與服務程式庫。文件
LiteLLM 提供者路由hosted_vllm/（用於 OpenAI 相容伺服器），vllm/（[已棄用]，用於 vLLM sdk 使用）
提供者文件vLLM ↗
支援的端點/chat/completions、/embeddings、/completions、/rerank、/audio/transcriptions

快速開始​

使用 - litellm.completion（呼叫 OpenAI 相容端點）​

vLLM 提供 OpenAI 相容端點 - 以下是如何透過 LiteLLM 呼叫它

若要使用 litellm 呼叫代管的 vllm 伺服器，請將下列內容加入您的 completion 呼叫中

  • model="hosted_vllm/<your-vllm-model-name>"
  • api_base = "your-hosted-vllm-server"
import litellm 

response = litellm.completion(
            model="hosted_vllm/facebook/opt-125m", # pass the vllm model name
            messages=messages,
            api_base="https://hosted-vllm-api.co",
            temperature=0.2,
            max_tokens=80)

print(response)

使用 - LiteLLM Proxy Server（呼叫 OpenAI 相容端點）​

以下是如何使用 LiteLLM Proxy Server 呼叫 OpenAI 相容端點

  1. 修改 config.yaml
model_list:
  - model_name: my-model
    litellm_params:
      model: hosted_vllm/facebook/opt-125m  # add hosted_vllm/ prefix to route as OpenAI provider
      api_base: https://hosted-vllm-api.co      # add api base for OpenAI compatible provider
  1. 啟動 proxy
$ litellm --config /path/to/config.yaml
  1. 將請求送至 LiteLLM Proxy Server
import openai
client = openai.OpenAI(
    api_key="sk-<your-litellm-api-key>",             # pass litellm proxy key, if you're using virtual keys
    base_url="http://0.0.0.0:4000" # litellm-proxy-base url
)

response = client.chat.completions.create(
    model="my-model",
    messages = [
        {
            "role": "user",
            "content": "what llm are you"
        }
    ],
)

print(response)

Reasoning Effort​

from litellm import completion

response = completion(
    model="hosted_vllm/gpt-oss-120b",
    messages=[{"role": "user", "content": "whats 2 + 2"}],
    reasoning_effort="high",
    api_base="https://hosted-vllm-api.co",
)
print(response)

嵌入​

vLLM 提供 OpenAI 相容的 /v1/embeddings。當用戶端省略 encoding_format 時，LiteLLM 會為 OpenAI 相容的 embedding 路由預設設定它（request → model litellm_params → LITELLM_DEFAULT_EMBEDDING_ENCODING_FORMAT → float）。請參閱 Embeddings。

from litellm import embedding   
import os

os.environ["HOSTED_VLLM_API_BASE"] = "http://localhost:8000"


embedding = embedding(model="hosted_vllm/facebook/opt-125m", input=["Hello world"])

print(embedding)

重排​

from litellm import rerank
import os

os.environ["HOSTED_VLLM_API_BASE"] = "http://localhost:8000"
os.environ["HOSTED_VLLM_API_KEY"] = ""  # [optional], if your VLLM server requires an API key

query = "What is the capital of the United States?"
documents = [
    "Carson City is the capital city of the American state of Nevada.",
    "The Commonwealth of the Northern Mariana Islands is a group of islands in the Pacific Ocean. Its capital is Saipan.",
    "Washington, D.C. is the capital of the United States.",
    "Capital punishment has existed in the United States since before it was a country.",
]

response = rerank(
    model="hosted_vllm/your-rerank-model",
    query=query,
    documents=documents,
    top_n=3,
)
print(response)

非同步使用​

from litellm import arerank
import os, asyncio

os.environ["HOSTED_VLLM_API_BASE"] = "http://localhost:8000"
os.environ["HOSTED_VLLM_API_KEY"] = ""  # [optional], if your VLLM server requires an API key

async def test_async_rerank(): 
    query = "What is the capital of the United States?"
    documents = [
        "Carson City is the capital city of the American state of Nevada.",
        "The Commonwealth of the Northern Mariana Islands is a group of islands in the Pacific Ocean. Its capital is Saipan.",
        "Washington, D.C. is the capital of the United States.",
        "Capital punishment has existed in the United States since before it was a country.",
    ]

    response = await arerank(
        model="hosted_vllm/your-rerank-model",
        query=query,
        documents=documents,
        top_n=3,
    )
    print(response)

asyncio.run(test_async_rerank())

截斷長文件​

vLLM 的 /rerank 端點具有自己的截斷控制，而 LiteLLM 在設定後會將其轉發至 hosted_vllm rerank 模型。若沒有其中之一，超過 reranker context 視窗的文件會因 context 長度錯誤而失敗

參數型別描述
truncate_prompt_tokensinteger在評分前，將每組查詢與文件配對截斷為這麼多 tokens
truncation_sideleft 或 right輸入內容要截斷的端點
max_tokens_per_docinteger將每份文件上限設為這麼多 tokens
max_tokens_per_queryinteger將查詢上限設為這麼多 tokens

無效值（例如 truncation_side: "middle"）會在送往 vLLM 之前先回傳 400

from litellm import rerank

response = rerank(
    model="hosted_vllm/your-rerank-model",
    query="What is the capital of the United States?",
    documents=["<a document longer than the reranker context window>"],
    truncate_prompt_tokens=512,
    truncation_side="left",
)
print(response)

圖片編輯​

vLLM-Omni 針對 OpenAI 相容的 /v1/images/edits 提供影像編輯模型，例如 Qwen/Qwen-Image-Edit-2511。請使用 hosted_vllm/ 前綴，並將 api_base 指向 vLLM-Omni 伺服器；其他提供者欄位，例如 seed 或 negative_prompt，會作為表單欄位傳遞。vLLM-Omni 沒有 quality 或 input_fidelity 表單欄位，並且接受其遮罩為 mask_image（URL 字串），而不是 OpenAI 的 mask 檔案，因此 LiteLLM 會拒絕 mask、quality 和 input_fidelity 用於 hosted_vllm/ 模型，除非設定了 drop_params: true；在此情況下，它們會在請求送出前被捨棄。若要遮罩編輯，請將 mask_image 作為額外欄位，並提供遮罩的 URL。

from litellm import image_edit
import os

os.environ["HOSTED_VLLM_API_BASE"] = "http://localhost:8091"

response = image_edit(
    model="hosted_vllm/Qwen/Qwen-Image-Edit-2511",
    image=open("original_image.png", "rb"),
    prompt="Add a red hat to the person in the image",
)

print(response)

將影片 URL 傳送至 VLLM​

VLLM 的範例實作 請見此處

使用此方式可用 OpenAI 的 files 訊息類型，以相同格式將影片 URL 傳送至 VLLM + Gemini。

將影片 URL 傳送至 VLLM 有兩種方式：

  1. 直接傳遞影片 URL
{"type": "file", "file": {"file_id": video_url}},
  1. 將影片資料以 base64 傳遞
{"type": "file", "file": {"file_data": f"data:video/mp4;base64,{video_data_base64}"}}
from litellm import completion

messages=[
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "Summarize the following video"
            },
            {
                "type": "file",
                "file": {
                    "file_id": "https://www.youtube.com/watch?v=dQw4w9WgXcQ"
                }
            }
        ]
    }
]

# call vllm 
os.environ["HOSTED_VLLM_API_BASE"] = "https://hosted-vllm-api.co"
os.environ["HOSTED_VLLM_API_KEY"] = "" # [optional], if your VLLM server requires an API key
response = completion(
    model="hosted_vllm/qwen", # pass the vllm model name
    messages=messages,
)

# call gemini 
os.environ["GEMINI_API_KEY"] = "your-gemini-api-key"
response = completion(
    model="gemini/gemini-3.8-flash", # pass the gemini model name
    messages=messages,
)

print(response)

（已棄用）用於套件化 vllm 安裝​

使用 - litellm.completion​

uv add litellm vllm
import litellm 

response = litellm.completion(
            model="vllm/facebook/opt-125m", # add a vllm prefix so litellm knows the custom_llm_provider==vllm
            messages=messages,
            temperature=0.2,
            max_tokens=80)

print(response)

批次完成​

from litellm import batch_completion

model_name = "facebook/opt-125m"
provider = "vllm"
messages = [[{"role": "user", "content": "Hey, how's it going"}] for _ in range(5)]

response_list = batch_completion(
            model=model_name, 
            custom_llm_provider=provider, # can easily switch to huggingface, replicate, together ai, sagemaker, etc.
            messages=messages,
            temperature=0.2,
            max_tokens=80,
        )
print(response_list)

提示詞樣板​

對於具有特殊提示詞樣板的模型（例如 Llama2），我們會將提示詞格式化以符合其樣板。

如果我們不支援您需要的模型怎麼辦？ 您也可以指定自己的自訂提示詞格式化方式，以防我們尚未涵蓋您的模型。

這是否表示您必須為所有模型指定提示詞？ 不用。預設情況下，我們會將您的訊息內容串接成提示詞（適用於 Bloom、T-5、Llama-2 base models 等預期格式）

預設提示詞樣板

def default_pt(messages):
    return " ".join(message["content"] for message in messages)

LiteLLM 中提示詞樣板運作方式的程式碼

我們已經有提示詞樣板的模型​

模型名稱Works for Models函式呼叫
meta-llama/Llama-2-7b-chatAll meta-llama llama2 chat modelscompletion(model='vllm/meta-llama/Llama-2-7b', messages=messages, api_base="your_api_endpoint")
tiiuae/falcon-7b-instructAll falcon instruct modelscompletion(model='vllm/tiiuae/falcon-7b-instruct', messages=messages, api_base="your_api_endpoint")
mosaicml/mpt-7b-chatAll mpt chat modelscompletion(model='vllm/mosaicml/mpt-7b-chat', messages=messages, api_base="your_api_endpoint")
codellama/CodeLlama-34b-Instruct-hfAll codellama instruct modelscompletion(model='vllm/codellama/CodeLlama-34b-Instruct-hf', messages=messages, api_base="your_api_endpoint")
WizardLM/WizardCoder-Python-34B-V1.0All wizardcoder modelscompletion(model='vllm/WizardLM/WizardCoder-Python-34B-V1.0', messages=messages, api_base="your_api_endpoint")
Phind/Phind-CodeLlama-34B-v2All phind-codellama modelscompletion(model='vllm/Phind/Phind-CodeLlama-34B-v2', messages=messages, api_base="your_api_endpoint")

自訂提示詞樣板​

# Create your own custom prompt template works 
litellm.register_prompt_template(
	model="togethercomputer/LLaMA-2-7B-32K",
	roles={
            "system": {
                "pre_message": "[INST] <<SYS>>\n",
                "post_message": "\n<</SYS>>\n [/INST]\n"
            },
            "user": { 
                "pre_message": "[INST] ",
                "post_message": " [/INST]\n"
            }, 
            "assistant": {
                "pre_message": "\n",
                "post_message": "\n",
            }
        } # tell LiteLLM how you want to map the openai messages to this model
)

def test_vllm_custom_model():
    model = "vllm/togethercomputer/LLaMA-2-7B-32K"
    response = completion(model=model, messages=messages)
    print(response['choices'][0]['message']['content'])
    return response

test_vllm_custom_model()

實作程式碼

LiteLLM Enterprise
為正式環境打造的 SSO/SAML、稽核記錄、支出追蹤、多團隊管理與防護欄。
深入瞭解 →