Current section

Files

Jump to
cmdc lib cmdc plugin builtin recovery.ex
Raw

lib/cmdc/plugin/builtin/recovery.ex

defmodule CMDC.Plugin.Builtin.Recovery do
@moduledoc """
P1 异常恢复策略链插件 — 工具失败时按策略链逐级恢复。
策略链(按顺序执行):
1. **重试**:相同参数重试(可配次数 + exponential backoff)
2. **切换模型**:失败次数超阈值后 intervene 切换到备用模型
3. **人工升级**:emit 告警事件通知管理员
4. **中止**:策略全部耗尽后 abort + 生成故障报告
## 配置
{CMDC.Plugin.Builtin.Recovery,
max_retries: 3, # 最大重试次数
backoff_base_ms: 1000, # 退避基数(毫秒)
fallback_model: "qwen3-max", # 备用模型(nil 则跳过模型切换)
escalation_event: true # 是否 emit 升级事件
}
## 触发 Hook
`{:on_tool_error, name, call_id, error, attempt}` — 工具执行失败时触发。
"""
@behaviour CMDC.Plugin
require Logger
@default_max_retries 3
@default_backoff_base_ms 1000
# ==========================================================================
# Callbacks
# ==========================================================================
@impl true
def init(opts) do
state = %{
max_retries: Keyword.get(opts, :max_retries, @default_max_retries),
backoff_base_ms: Keyword.get(opts, :backoff_base_ms, @default_backoff_base_ms),
fallback_model: Keyword.get(opts, :fallback_model),
escalation_event: Keyword.get(opts, :escalation_event, true),
model_switched: false
}
{:ok, state}
end
@impl true
def priority, do: 150
@impl true
def handle_event({:on_tool_error, name, _call_id, error, attempt}, state, _ctx) do
cond do
attempt < state.max_retries ->
backoff_ms = (state.backoff_base_ms * :math.pow(2, attempt - 1)) |> round()
Logger.info("[Recovery] 工具 #{name} 失败(第 #{attempt} 次),#{backoff_ms}ms 后重试")
Process.sleep(backoff_ms)
{:continue, state}
state.fallback_model != nil and not state.model_switched ->
Logger.warning("[Recovery] 工具 #{name} 重试耗尽,切换到备用模型 #{state.fallback_model}")
{:intervene,
"工具 #{name} 多次失败(错误: #{inspect(error)})。已切换到备用模型 #{state.fallback_model},请用不同的方法重试。",
%{state | model_switched: true}}
state.escalation_event ->
Logger.error("[Recovery] 工具 #{name} 恢复策略耗尽,上报人工处理")
{:emit,
{:recovery_escalation,
%{tool: name, error: inspect(error), attempt: attempt, action: :escalated}}, state}
true ->
Logger.error("[Recovery] 工具 #{name} 恢复失败,中止 Agent")
{:abort, "工具 #{name} 多次失败且恢复策略耗尽: #{inspect(error)}", state}
end
end
def handle_event(_event, state, _ctx), do: {:continue, state}
end