---
title: "Triton Inference Server"
url: "/docs/providers/triton-inference-server"
canonical_url: "https://docs.litellm.ai/docs/providers/triton-inference-server"
type: "docs"
last_updated: "2026-10-08"
related:
  - "/docs/providers/topaz"
  - "/docs/providers/v0"
---
# Triton Inference Server

> Index of all LiteLLM docs: https://docs.litellm.ai/llms.txt


LiteLLM 支援 Triton Inference Server 上的嵌入模型

| 屬性 | 詳細資料 |
|-------|-------|
| 說明 | NVIDIA Triton Inference Server |
| LiteLLM 上的提供者路由 | `triton/` |
| 支援的操作 | `/chat/completion`, `/completion`, `/embedding` |
| 支援的 Triton 端點 | `/infer`, `/generate`, `/embeddings` |
| 提供者文件連結 | [Triton Inference Server ↗](https://developer.nvidia.com/triton-inference-server) |

## Triton `/generate` - 聊天完成 {#triton-generate---chat-completion}

**SDK**

使用 `triton/` 前綴來路由至 triton server
```python
from litellm import completion
response = completion(
    model="triton/llama-3-8b-instruct",
    messages=[{"role": "user", "content": "who are u?"}],
    max_tokens=10,
    api_base="http://localhost:8000/generate",
)
```

**PROXY**

1. 將模型新增至您的 config.yaml

  ```yaml
  model_list:
    - model_name: my-triton-model
      litellm_params:
        model: triton/<your-triton-model>"
        api_base: https://your-triton-api-base/triton/generate
  ```

2. 啟動 proxy 

  ```bash
  $ litellm --config /path/to/config.yaml --detailed_debug
  ```

3. 將請求送至 LiteLLM Proxy Server

**OpenAI Python v1.0.0+**

    ```python
    import openai
    from openai import OpenAI

    # set base_url to your proxy server
    # set api_key to send to proxy server
    client = OpenAI(api_key="<proxy-api-key>", base_url="http://0.0.0.0:4000")

    response = client.chat.completions.create(
        model="my-triton-model",
        messages=[{"role": "user", "content": "who are u?"}],
        max_tokens=10,
    )

    print(response)

    ```

**curl**

  `--header` 為選用項目，僅在您使用帶有 Virtual Keys 的 litellm proxy 時才需要

    ```shell
    curl --location 'http://0.0.0.0:4000/chat/completions' \
    --header 'Content-Type: application/json' \
    --header "Authorization: Bearer $LITELLM_API_KEY" \
    --data ' {
    "model": "my-triton-model",
    "messages": [{"role": "user", "content": "who are u?"}]
    }'

    ```

## Triton `/infer` - 聊天完成 {#triton-infer---chat-completion}

**SDK**

使用 `triton/` 前綴來路由至 triton server
```python
from litellm import completion

response = completion(
    model="triton/llama-3-8b-instruct",
    messages=[{"role": "user", "content": "who are u?"}],
    max_tokens=10,
    api_base="http://localhost:8000/infer",
)
```

**PROXY**

1. 將模型新增至您的 config.yaml

  ```yaml
  model_list:
    - model_name: my-triton-model
      litellm_params:
        model: triton/<your-triton-model>"
        api_base: https://your-triton-api-base/triton/infer
  ```

2. 啟動 proxy 

  ```bash
  $ litellm --config /path/to/config.yaml --detailed_debug
  ```

3. 將請求送至 LiteLLM Proxy Server

**OpenAI Python v1.0.0+**

    ```python
    import openai
    from openai import OpenAI

    # set base_url to your proxy server
    # set api_key to send to proxy server
    client = OpenAI(api_key="<proxy-api-key>", base_url="http://0.0.0.0:4000")

    response = client.chat.completions.create(
        model="my-triton-model",
        messages=[{"role": "user", "content": "who are u?"}],
        max_tokens=10,
    )

    print(response)

    ```

**curl**

  `--header` 為選用項目，僅在您使用帶有 Virtual Keys 的 litellm proxy 時才需要

    ```shell
    curl --location 'http://0.0.0.0:4000/chat/completions' \
    --header 'Content-Type: application/json' \
    --header "Authorization: Bearer $LITELLM_API_KEY" \
    --data ' {
    "model": "my-triton-model",
    "messages": [{"role": "user", "content": "who are u?"}]
    }'

    ```

## Triton `/embeddings` - 嵌入 {#triton-embeddings---embedding}

**SDK**

使用 `triton/` 前綴來路由至 triton server
```python
import litellm
import os

response = await litellm.aembedding(
    model="triton/<your-triton-model>",                                                       
    api_base="https://your-triton-api-base/triton/embeddings", # /embeddings endpoint you want litellm to call on your server
    input=["good morning from litellm"],
)
```

**PROXY**

1. 將模型新增至您的 config.yaml

  ```yaml
  model_list:
    - model_name: my-triton-model
      litellm_params:
        model: triton/<your-triton-model>"
        api_base: https://your-triton-api-base/triton/embeddings
  ```

2. 啟動 proxy 

  ```bash
  $ litellm --config /path/to/config.yaml --detailed_debug
  ```

3. 將請求送至 LiteLLM Proxy Server

**OpenAI Python v1.0.0+**

    ```python
    import openai
    from openai import OpenAI

    # set base_url to your proxy server
    # set api_key to send to proxy server
    client = OpenAI(api_key="<proxy-api-key>", base_url="http://0.0.0.0:4000")

    response = client.embeddings.create(
        input=["hello from litellm"],
        model="my-triton-model"
    )

    print(response)

    ```

**curl**

  `--header` 為選用項目，僅在您使用帶有 Virtual Keys 的 litellm proxy 時才需要

    ```shell
    curl --location 'http://0.0.0.0:4000/embeddings' \
    --header 'Content-Type: application/json' \
    --header "Authorization: Bearer $LITELLM_API_KEY" \
    --data ' {
    "model": "my-triton-model",
    "input": ["write a litellm poem"]
    }'

    ```

## Related pages

- [Topaz](https://docs.litellm.ai/docs/providers/topaz.md)
- [v0](https://docs.litellm.ai/docs/providers/v0.md)