使用 completion() 搭配備援（Failover）以提升可靠性

本教學示範如何使用 completion() 函式搭配模型備援（也稱為 failover）來確保可靠性。LLM API 可能不穩定，搭配備援的 completion() 可確保您的請求總是能取得回應

為虛擬金鑰設定備援

使用方式

若要在 completion() 中使用備援模型，請在 fallbacks 參數中指定模型清單。

fallbacks 清單應包含您要使用的主要模型，接著再加入可在主要模型無法提供回應時作為備援的其他模型。

response = completion(model="bad-model", fallbacks=["gpt-3.5-turbo" "command-nightly"], messages=messages)

completion_with_fallbacks() 的運作方式

completion_with_fallbacks() 函式會以在 completion(model=model) 中指定為 model 的主要模型嘗試進行 completion 請求。若主要模型失敗或發生錯誤，系統會依指定順序自動嘗試 fallbacks 模型。這可確保即使主要模型無法使用，也能取得回應。

請求的輸出

Completion with 'bad-model': got exception Unable to map your input to a model. Check your input - {'model': 'bad-model'



completion call gpt-3.5-turbo
{
  "id": "chatcmpl-7qTmVRuO3m3gIBg4aTmAumV1TmQhB",
  "object": "chat.completion",
  "created": 1692741891,
  "model": "gpt-3.5-turbo-0613",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "I apologize, but as an AI, I do not have the capability to provide real-time weather updates. However, you can easily check the current weather in San Francisco by using a search engine or checking a weather website or app."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 16,
    "completion_tokens": 46,
    "total_tokens": 62
  }
}

Model Fallbacks 實作的主要元件：

• 透過 fallbacks 進行迴圈
• 針對受速率限制的模型進行冷卻時間

透過 fallbacks 進行迴圈

允許每個請求有 45seconds。在這 45 秒內，此函式會嘗試呼叫設定為 model 的主要模型。若模型失敗，則會依序循環呼叫備用的 fallbacks 模型，並嘗試在此處設定的分配 45s 時間內取得回應：

while response == None and time.time() - start_time < 45:
        for model in fallbacks:

針對受速率限制的模型進行冷卻時間

如果模型 API 請求導致錯誤，請讓它冷卻 60s

except Exception as e:
  print(f"got exception {e} for model {model}")
  rate_limited_models.add(model)
  model_expiration_times[model] = (
      time.time() + 60
  )  # cool down this selected model
  pass

在發出 LLM API 請求之前，我們會檢查所選模型是否處於 rate_limited_models，若是，則略過發出 API 請求

if (
  model in rate_limited_models
):  # check if model is currently cooling down
  if (
      model_expiration_times.get(model)
      and time.time() >= model_expiration_times[model]
  ):
      rate_limited_models.remove(
          model
      )  # check if it's been 60s of cool down and remove model
  else:
      continue  # skip model

含備援的 completion() 完整程式碼

    response = None
    rate_limited_models = set()
    model_expiration_times = {}
    start_time = time.time()
    fallbacks = [kwargs["model"]] + kwargs["fallbacks"]
    del kwargs["fallbacks"]  # remove fallbacks so it's not recursive

    while response == None and time.time() - start_time < 45:
        for model in fallbacks:
            # loop thru all models
            try:
                if (
                    model in rate_limited_models
                ):  # check if model is currently cooling down
                    if (
                        model_expiration_times.get(model)
                        and time.time() >= model_expiration_times[model]
                    ):
                        rate_limited_models.remove(
                            model
                        )  # check if it's been 60s of cool down and remove model
                    else:
                        continue  # skip model

                # delete model from kwargs if it exists
                if kwargs.get("model"):
                    del kwargs["model"]

                print("making completion call", model)
                response = litellm.completion(**kwargs, model=model)

                if response != None:
                    return response

            except Exception as e:
                print(f"got exception {e} for model {model}")
                rate_limited_models.add(model)
                model_expiration_times[model] = (
                    time.time() + 60
                )  # cool down this selected model
                pass
    return response