[舊版 Proxy 👉 新版 proxy 在這裡] 本機 LiteLLM Proxy Server
一個快速且輕量、相容 OpenAI 的伺服器，可呼叫 100+ 個 LLM API。
資訊
文件已過時。新文件 👉 這裡
使用方式
uv tool install 'litellm[proxy]'
$ litellm --model ollama/codellama
#INFO: Ollama running on http://0.0.0.0:4000
測試
在新的 shell 中執行：
$ litellm --test
取代 openai base
import openai
openai.api_base = "http://0.0.0.0:4000"
print(openai.ChatCompletion.create(model="test", messages=[{"role":"user", "content":"Hey!"}]))
其他支援的模型：
- VLLM
- OpenAI Compatible Server
- Huggingface
- Anthropic
- TogetherAI
- Replicate
- Petals
- Palm
- Azure OpenAI
- AI21
- Cohere
假設您正在本機執行 vllm
$ litellm --model vllm/facebook/opt-125m
$ litellm --model openai/<model_name> --api_base <your-api-base>
$ export HUGGINGFACE_API_KEY=my-api-key #[OPTIONAL]
$ litellm --model claude-sonnet-5
$ export ANTHROPIC_API_KEY=my-api-key
$ litellm --model claude-sonnet-5
$ export TOGETHERAI_API_KEY=my-api-key
$ litellm --model together_ai/lmsys/vicuna-13b-v1.5-16k
$ export REPLICATE_API_KEY=my-api-key
$ litellm \
--model replicate/meta/llama-2-70b-chat:02e509c789964a7ea8736978a43525956ef40397be9033abf9fd2badfe68c9e3
$ litellm --model petals/meta-llama/Llama-2-70b-chat-hf
$ export PALM_API_KEY=my-palm-key
$ litellm --model palm/chat-bison
$ export AZURE_API_KEY=my-api-key
$ export AZURE_API_BASE=my-api-base
$ litellm --model azure/my-deployment-name
$ export AI21_API_KEY=my-api-key
$ litellm --model j2-light
$ export COHERE_API_KEY=my-api-key
$ litellm --model command-nightly
教學：搭配 Multiple LLMs + LibreChat/Chatbot-UI/Auto-Gen/ChatDev/Langroid 等使用
- Multiple LLMs
- LibreChat
- SmartChatbotUI
- AutoGen
- AutoGen Multi-LLM
- ChatDev
- Langroid
取代 openai base：
import openai
openai.api_key = "any-string-here"
openai.api_base = "http://0.0.0.0:4000" # your proxy url
# call openai
response = openai.ChatCompletion.create(model="gpt-5.6-luna", messages=[{"role": "user", "content": "Hey"}])
print(response)
# call cohere
response = openai.ChatCompletion.create(model="command-nightly", messages=[{"role": "user", "content": "Hey"}])
print(response)
1. 複製儲存庫
git clone https://github.com/danny-avila/LibreChat.git
2. 修改 docker-compose.yml
OPENAI_REVERSE_PROXY=http://host.docker.internal:4000/v1/chat/completions
3. 將假的 OpenAI key 存入 .env
OPENAI_API_KEY=sk-<your-api-key>
4. 執行 LibreChat：
docker compose up
1. 複製儲存庫
git clone https://github.com/dotneet/smart-chatbot-ui.git
2. 安裝相依性
npm i
3. 建立您的環境
cp .env.local.example .env.local
4. 設定 API 金鑰與 Base
OPENAI_API_KEY="my-fake-key"
OPENAI_API_HOST="http://0.0.0.0:4000
5. 使用 docker compose 執行
docker compose up -d
uv add pyautogen
from autogen import AssistantAgent, UserProxyAgent, oai
config_list=[
{
"model": "my-fake-model",
"api_base": "http://0.0.0.0:4000", #litellm compatible endpoint
"api_type": "open_ai",
"api_key": "NULL", # just a placeholder
}
]
response = oai.Completion.create(config_list=config_list, prompt="Hi")
print(response) # works fine
llm_config={
"config_list": config_list,
}
assistant = AssistantAgent("assistant", llm_config=llm_config)
user_proxy = UserProxyAgent("user_proxy")
user_proxy.initiate_chat(assistant, message="Plot a chart of META and TESLA stock price change YTD.", config_list=config_list)
本教學感謝 @victordibia。
from autogen import AssistantAgent, GroupChatManager, UserProxyAgent
from autogen.agentchat import GroupChat
config_list = [
{
"model": "ollama/mistralorca",
"api_base": "http://0.0.0.0:4000", # litellm compatible endpoint
"api_type": "open_ai",
"api_key": "NULL", # just a placeholder
}
]
llm_config = {"config_list": config_list, "seed": 42}
code_config_list = [
{
"model": "ollama/phind-code",
"api_base": "http://0.0.0.0:4000", # litellm compatible endpoint
"api_type": "open_ai",
"api_key": "NULL", # just a placeholder
}
]
code_config = {"config_list": code_config_list, "seed": 42}
admin = UserProxyAgent(
name="Admin",
system_message="A human admin. Interact with the planner to discuss the plan. Plan execution needs to be approved by this admin.",
llm_config=llm_config,
code_execution_config=False,
)
engineer = AssistantAgent(
name="Engineer",
llm_config=code_config,
system_message="""Engineer. You follow an approved plan. You write python/shell code to solve tasks. Wrap the code in a code block that specifies the script type. The user can't modify your code. So do not suggest incomplete code which requires others to modify. Don't use a code block if it's not intended to be executed by the executor.
Don't include multiple code blocks in one response. Do not ask others to copy and paste the result. Check the execution result returned by the executor.
If the result indicates there is an error, fix the error and output the code again. Suggest the full code instead of partial code or code changes. If the error can't be fixed or if the task is not solved even after the code is executed successfully, analyze the problem, revisit your assumption, collect additional info you need, and think of a different approach to try.
""",
)
planner = AssistantAgent(
name="Planner",
system_message="""Planner. Suggest a plan. Revise the plan based on feedback from admin and critic, until admin approval.
The plan may involve an engineer who can write code and a scientist who doesn't write code.
Explain the plan first. Be clear which step is performed by an engineer, and which step is performed by a scientist.
""",
llm_config=llm_config,
)
executor = UserProxyAgent(
name="Executor",
system_message="Executor. Execute the code written by the engineer and report the result.",
human_input_mode="NEVER",
llm_config=llm_config,
code_execution_config={"last_n_messages": 3, "work_dir": "paper"},
)
critic = AssistantAgent(
name="Critic",
system_message="Critic. Double check plan, claims, code from other agents and provide feedback. Check whether the plan includes adding verifiable info such as source URL.",
llm_config=llm_config,
)
groupchat = GroupChat(
agents=[admin, engineer, planner, executor, critic],
messages=[],
max_round=50,
)
manager = GroupChatManager(groupchat=groupchat, llm_config=llm_config)
admin.initiate_chat(
manager,
message="""
""",
)
本教學感謝 @Nathan。
設定 ChatDev (文件)
git clone https://github.com/OpenBMB/ChatDev.git
cd ChatDev
conda create -n ChatDev_conda_env python=3.13 -y
conda activate ChatDev_conda_env
uv add -r requirements.txt
使用 Proxy 執行 ChatDev
export OPENAI_API_KEY="sk-<your-api-key>"
export OPENAI_BASE_URL="http://0.0.0.0:4000"
python3 run.py --task "a script that says hello world" --name "hello world"
uv add langroid
from langroid.language_models.openai_gpt import OpenAIGPTConfig, OpenAIGPT
# configure the LLM
my_llm_config = OpenAIGPTConfig(
# where proxy server is listening
api_base="http://0.0.0.0:4000",
)
# create llm, one-off interaction
llm = OpenAIGPT(my_llm_config)
response = mdl.chat("What is the capital of China?", max_tokens=50)
# Create an Agent with this LLM, wrap it in a Task, and
# run it as an interactive chat app:
from langroid.agent.base import ChatAgent, ChatAgentConfig
from langroid.agent.task import Task
agent_config = ChatAgentConfig(llm=my_llm_config, name="my-llm-agent")
agent = ChatAgent(agent_config)
task = Task(agent, name="my-llm-task")
task.run()
本教學感謝 @pchalasani 與 Langroid。
本機 Proxy
以下說明如何使用本機 proxy，來測試不同 github repo 的 codellama/mistral 等模型
uv add litellm
$ ollama pull codellama # OUR Local CodeLlama
$ litellm --model ollama/codellama --temperature 0.3 --max_tokens 2048
教學：搭配 Multiple LLMs + Aider/AutoGen/Langroid 等使用
- Multiple LLMs
- ContinueDev
- Aider
- AutoGen
- AutoGen Multi-LLM
- ChatDev
- Langroid
- GPT-Pilot
- guidance
$ litellm
#INFO: litellm proxy running on http://0.0.0.0:4000
將請求送至您的 proxy
import openai
openai.api_key = "any-string-here"
openai.api_base = "http://0.0.0.0:4000" # your proxy url
# call gpt-5.6-luna
response = openai.ChatCompletion.create(model="gpt-5.6-luna", messages=[{"role": "user", "content": "Hey"}])
print(response)
# call ollama/llama2
response = openai.ChatCompletion.create(model="ollama/llama2", messages=[{"role": "user", "content": "Hey"}])
print(response)
Continue-Dev 將 ChatGPT 帶入 VSCode。請參閱這裡的安裝方式。
在 config.py 中，將其設為您的預設模型。
default=OpenAI(
api_key="IGNORED",
model="fake-model-name",
context_length=2048, # customize if needed for your model
api_base="http://localhost:4000" # your proxy server url
),
本教學感謝 @vividfog。
$ uv add aider
$ aider --openai-api-base http://0.0.0.0:4000 --openai-api-key fake-key
uv add pyautogen
from autogen import AssistantAgent, UserProxyAgent, oai
config_list=[
{
"model": "my-fake-model",
"api_base": "http://localhost:4000", #litellm compatible endpoint
"api_type": "open_ai",
"api_key": "NULL", # just a placeholder
}
]
response = oai.Completion.create(config_list=config_list, prompt="Hi")
print(response) # works fine
llm_config={
"config_list": config_list,
}
assistant = AssistantAgent("assistant", llm_config=llm_config)
user_proxy = UserProxyAgent("user_proxy")
user_proxy.initiate_chat(assistant, message="Plot a chart of META and TESLA stock price change YTD.", config_list=config_list)
本教學感謝 @victordibia。
from autogen import AssistantAgent, GroupChatManager, UserProxyAgent
from autogen.agentchat import GroupChat
config_list = [
{
"model": "ollama/mistralorca",
"api_base": "http://localhost:4000", # litellm compatible endpoint
"api_type": "open_ai",
"api_key": "NULL", # just a placeholder
}
]
llm_config = {"config_list": config_list, "seed": 42}
code_config_list = [
{
"model": "ollama/phind-code",
"api_base": "http://localhost:4000", # litellm compatible endpoint
"api_type": "open_ai",
"api_key": "NULL", # just a placeholder
}
]
code_config = {"config_list": code_config_list, "seed": 42}
admin = UserProxyAgent(
name="Admin",
system_message="A human admin. Interact with the planner to discuss the plan. Plan execution needs to be approved by this admin.",
llm_config=llm_config,
code_execution_config=False,
)
engineer = AssistantAgent(
name="Engineer",
llm_config=code_config,
system_message="""Engineer. You follow an approved plan. You write python/shell code to solve tasks. Wrap the code in a code block that specifies the script type. The user can't modify your code. So do not suggest incomplete code which requires others to modify. Don't use a code block if it's not intended to be executed by the executor.
Don't include multiple code blocks in one response. Do not ask others to copy and paste the result. Check the execution result returned by the executor.
If the result indicates there is an error, fix the error and output the code again. Suggest the full code instead of partial code or code changes. If the error can't be fixed or if the task is not solved even after the code is executed successfully, analyze the problem, revisit your assumption, collect additional info you need, and think of a different approach to try.
""",
)
planner = AssistantAgent(
name="Planner",
system_message="""Planner. Suggest a plan. Revise the plan based on feedback from admin and critic, until admin approval.
The plan may involve an engineer who can write code and a scientist who doesn't write code.
Explain the plan first. Be clear which step is performed by an engineer, and which step is performed by a scientist.
""",
llm_config=llm_config,
)
executor = UserProxyAgent(
name="Executor",
system_message="Executor. Execute the code written by the engineer and report the result.",
human_input_mode="NEVER",
llm_config=llm_config,
code_execution_config={"last_n_messages": 3, "work_dir": "paper"},
)
critic = AssistantAgent(
name="Critic",
system_message="Critic. Double check plan, claims, code from other agents and provide feedback. Check whether the plan includes adding verifiable info such as source URL.",
llm_config=llm_config,
)
groupchat = GroupChat(
agents=[admin, engineer, planner, executor, critic],
messages=[],
max_round=50,
)
manager = GroupChatManager(groupchat=groupchat, llm_config=llm_config)
admin.initiate_chat(
manager,
message="""
""",
)
本教學感謝 @Nathan。
設定 ChatDev (文件)
git clone https://github.com/OpenBMB/ChatDev.git
cd ChatDev
conda create -n ChatDev_conda_env python=3.13 -y
conda activate ChatDev_conda_env
uv add -r requirements.txt
使用 Proxy 執行 ChatDev
export OPENAI_API_KEY="sk-<your-api-key>"
export OPENAI_BASE_URL="http://0.0.0.0:4000"
python3 run.py --task "a script that says hello world" --name "hello world"
uv add langroid
from langroid.language_models.openai_gpt import OpenAIGPTConfig, OpenAIGPT
# configure the LLM
my_llm_config = OpenAIGPTConfig(
#format: "local/[URL where LiteLLM proxy is listening]
chat_model="local/localhost:4000",
chat_context_length=2048, # adjust based on model
)
# create llm, one-off interaction
llm = OpenAIGPT(my_llm_config)
response = mdl.chat("What is the capital of China?", max_tokens=50)
# Create an Agent with this LLM, wrap it in a Task, and
# run it as an interactive chat app:
from langroid.agent.base import ChatAgent, ChatAgentConfig
from langroid.agent.task import Task
agent_config = ChatAgentConfig(llm=my_llm_config, name="my-llm-agent")
agent = ChatAgent(agent_config)
task = Task(agent, name="my-llm-task")
task.run()
本教學感謝 @pchalasani 與 Langroid。
GPT-Pilot 協助您使用 AI 代理程式建立應用程式。更多資訊
在您的 .env 中，將 openai endpoint 設為您的本機伺服器。
OPENAI_ENDPOINT=http://0.0.0.0:4000
OPENAI_API_KEY=my-fake-key
一種用於控制大型語言模型的 guidance language。 https://github.com/guidance-ai/guidance
注意： Guidance 會傳送額外參數，例如 stop_sequences，若模型不支援，可能會導致某些模型失敗。
修正方式：使用 --drop_params 旗標啟動您的 proxy
litellm --model ollama/codellama --temperature 0.3 --max_tokens 2048 --drop_params
import guidance
# set api_base to your proxy
# set api_key to anything
gpt4 = guidance.llms.OpenAI("gpt-5.6-terra", api_base="http://0.0.0.0:4000", api_key="anything")
experts = guidance('''
{{#system~}}
You are a helpful and terse assistant.
{{~/system}}
{{#user~}}
I want a response to the following question:
{{query}}
Name 3 world-class experts (past or present) who would be great at answering this?
Don't answer the question yet.
{{~/user}}
{{#assistant~}}
{{gen 'expert_names' temperature=0 max_tokens=300}}
{{~/assistant}}
''', llm=gpt4)
result = experts(query='How can I be more productive?')
print(result)
備註
貢獻 您有在專案中使用這個伺服器嗎？請在這裡！貢獻您的教學
進階
記錄
Proxy 會將請求與回應記錄列印到 stdout。若要啟用請求層級記錄，請以 --debug 啟動；若要啟用詳細記錄，請以 --detailed_debug 啟動。若要保留記錄，請在您的設定中透過 litellm_settings.success_callback 將它們傳送到記錄整合；請參閱 記錄。
設定 Proxy
如果您需要：
- 儲存 API 金鑰
- 設定 litellm 參數（例如：捨棄未對應的參數、設定 fallback models 等）
- 設定特定模型參數（max tokens、temperature、api base、prompt template）
您可以只為該 session 設定這些內容（透過 cli），或透過設定檔讓這些設定在重新啟動後仍然保留。
儲存 API 金鑰
CLI 不會儲存 API 金鑰。請在啟動 proxy 之前將它們設定為環境變數，或透過 os.environ/ 從設定檔中參照它們：
$ export OPENAI_API_KEY=sk-...
$ litellm --model gpt-5.6-terra
model_list:
- model_name: gpt-5.6-terra
litellm_params:
model: openai/gpt-5.6-terra
api_key: os.environ/OPENAI_API_KEY
$ litellm --config config.yaml
LiteLLM Proxy 支援所有 litellm 支援的 api keys。請參閱 提供者，了解各提供者所需的環境變數，並參閱 Proxy 設定 取得完整的設定檔參考。
例如：設定 api base、max tokens 和 temperature。
針對該 session：
litellm --model ollama/llama2 \
--api_base http://localhost:11434 \
--max_tokens 250 \
--temperature 0.5
# OpenAI-compatible server running on http://0.0.0.0:4000
效能
我們使用 wrk 對 FastAPI server 進行了 500,000 個 HTTP 連線、持續 1 分鐘的負載測試。
結果如下：
Thread Stats Avg Stdev Max +/- Stdev
Latency 156.38ms 25.52ms 361.91ms 84.73%
Req/Sec 13.61 5.13 40.00 57.50%
383625 requests in 1.00m, 391.10MB read
Socket errors: connect 0, read 1632, write 1, timeout 0
支援／與創辦人聯絡
- 安排示範 👋
- 社群 Discord 💭
- 我們的電子郵件 ✉️ ishaan@berri.ai / krrish@berri.ai
