switchboard-100.pages.dev

simulate_policy

Project what a candidate routing policy would cost and deliver, WITHOUT applying it. Classes you omit keep their current routing. Definitions: deliveredRatePct is the chain-level probability that some link succeeds, i.e. 1 minus the product of every link failing — this, not a single model's requestSuccessRate, is what a class's minSuccessRate is judged against. expectedLatencyMs is the probability-weighted median across the chain. worstCaseLatencyMs is the SUM of p95 across the WHOLE chain — what a request pays if every link but the last fails. It is reported unconditionally, so it never understates the tail; read tailRiskProbability alongside it, which is the chance of getting past the primary at all. HOW A LATENCY CEILING IS JUDGED, because the two rules interact: a fallback whose own p95 exceeds the ceiling is a WARNING, not a blocker, since a slow answer beats no answer — so the class stays permissible. But the chain's summed worst case is checked too and raises a CHAIN_LATENCY warning when it exceeds the ceiling. So a chain can be permissible and still carry a flagged tail. Say which you mean: "the primary meets your 4s ceiling; the chain can reach Xms in the Y% of cases where the primary fails" is the honest phrasing. Cost counts every attempt, since a failed call is still billed. The candidate's compliance verdict is returned here too and is the same computation check_compliance runs, so you do not need both — call check_compliance only when you want to audit the LIVE policy. One side effect: this repaints the projection on the operator's screen so they can follow your reasoning, so iterate deliberately rather than sweeping dozens of candidates.

Input schema

{
  "type": "object",
  "properties": {
    "rules": {
      "type": "array",
      "description": "Candidate routing rules. One entry per traffic class you want to change; classes you omit keep their current rule.",
      "items": {
        "type": "object",
        "properties": {
          "classId": {
            "type": "string",
            "description": "Traffic class this rule routes."
          },
          "primaryModelId": {
            "type": "string",
            "description": "Model tried first, e.g. 'opencode-go/minimax-m3'."
          },
          "fallbackModelIds": {
            "type": "array",
            "items": {
              "type": "string"
            },
            "description": "Models tried in order if the primary fails. Pass [] for no fallback."
          }
        },
        "required": [
          "classId",
          "primaryModelId",
          "fallbackModelIds"
        ]
      }
    },
    "label": {
      "type": "string",
      "description": "Short human-readable name for this candidate, shown on screen."
    }
  },
  "required": [
    "rules"
  ]
}