基準測試 LLM
輕鬆透過檢視以下項目，對特定問題為 LLM 做基準測試：
- 回應
- 回應成本
- 回應時間
基準測試輸出
設定：
git clone https://github.com/BerriAI/litellm
cd 到 litellm/cookbook/benchmark 目錄
位置如下： https://github.com/BerriAI/litellm/tree/main/cookbook/benchmark
cd litellm/cookbook/benchmark
安裝相依套件
uv add litellm click tqdm tabulate termcolor
設定 - 在 benchmark.py 中設定 LLM API 金鑰 + LLM
在 benchmark/benchmark.py 中選擇您的 LLM、LLM API 金鑰與問題
支援的 LLM：https://docs.litellm.ai/docs/providers
# Define the list of models to benchmark
models = ['gpt-3.5-turbo', 'claude-2']
# Enter LLM API keys
os.environ['OPENAI_API_KEY'] = ""
os.environ['ANTHROPIC_API_KEY'] = ""
# List of questions to benchmark (replace with your questions)
questions = [
"When will BerriAI IPO?",
"When will LiteLLM hit $100M ARR?"
]
執行 benchmark.py
python3 benchmark.py
預期輸出
Running question: When will BerriAI IPO? for model: claude-2: 100%|████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:13<00:00, 4.41s/it]
Benchmark Results for 'When will BerriAI IPO?':
+-----------------+----------------------------------------------------------------------------------+---------------------------+------------+
| Model | Response | Response Time (seconds) | Cost ($) |
+=================+==================================================================================+===========================+============+
| gpt-3.5-turbo | As an AI language model, I cannot provide up-to-date information or predict | 1.55 seconds | $0.000122 |
| | future events. It is best to consult a reliable financial source or contact | | |
| | BerriAI directly for information regarding their IPO plans. | | |
+-----------------+----------------------------------------------------------------------------------+---------------------------+------------+
| togethercompute | I'm not able to provide information about future IPO plans or dates for BerriAI | 8.52 seconds | $0.000531 |
| r/llama-2-70b-c | or any other company. IPO (Initial Public Offering) plans and timelines are | | |
| hat | typically kept private by companies until they are ready to make a public | | |
| | announcement. It's important to note that IPO plans can change and are subject | | |
| | to various factors, such as market conditions, financial performance, and | | |
| | regulatory approvals. Therefore, it's difficult to predict with certainty when | | |
| | BerriAI or any other company will go public. If you're interested in staying | | |
| | up-to-date with BerriAI's latest news and developments, you may want to follow | | |
| | their official social media accounts, subscribe to their newsletter, or visit | | |
| | their website periodically for updates. | | |
+-----------------+----------------------------------------------------------------------------------+---------------------------+------------+
| claude-2 | I do not have any information about when or if BerriAI will have an initial | 3.17 seconds | $0.002084 |
| | public offering (IPO). As an AI assistant created by Anthropic to be helpful, | | |
| | harmless, and honest, I do not have insider knowledge about Anthropic's business | | |
| | plans or strategies. | | |
+-----------------+----------------------------------------------------------------------------------+---------------------------+------------+
支援
🤝 預約 1 對 1 會議： 與創辦人 Krrish 和 Ishaan 預約一場 1 對 1 會議，討論任何問題、提供意見回饋，或探索我們如何為您改進 LiteLLM。