switchboard-100.pages.dev

compare_models

Rank candidate models for one traffic class, using that class's own token profile and declared volume, so the money figures are comparable to each other and to get_routing_policy. Note this differs from the costPer1kSuccessfulUsd in list_models, which uses the measurement task's own token profile — do not quote the two side by side. Only models satisfying every HARD constraint of the class are ranked; the rest come back under rejected with the blocking codes. Ranking: 'cost' = lowest cost per delivered output; 'latency' = lowest measured p95; 'reliability' = highest success rate; 'balanced' (default) = costPerDelivered × (1 + p95ms/10000) × 1.5 if it fails quality gates. Fallbacks: performance constraints (quality gates, latency) soften to warnings for a fallback because a slow answer beats none, but governance constraints (retention, training) never soften — so a model rejected on RETENTION or TRAINING cannot serve this class in any position. Call this before proposing a routing change.

Input schema

{
  "type": "object",
  "properties": {
    "classId": {
      "type": "string",
      "description": "Traffic class id. Call list_traffic_classes for valid ids (e.g. 'realtime', 'batch', 'sensitive')."
    },
    "optimiseFor": {
      "type": "string",
      "enum": [
        "cost",
        "latency",
        "reliability",
        "balanced"
      ],
      "description": "Ranking objective. Default balanced."
    },
    "limit": {
      "type": "number",
      "description": "How many ranked candidates to return. Default 8."
    }
  },
  "required": [
    "classId"
  ]
}