Model compare

Compare models on the same prompt with POST /api/v1/compare or compare.create. The response includes each answer, ranking, latency and cost.

cURL

curl https://www.ninjachat.ai/api/v1/compare \
  -H "Authorization: Bearer $NINJACHAT_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: compare-2026-09-01-0001" \
  -d '{
    "messages": [{ "role": "user", "content": "Rewrite this release note for a non-technical audience: ..." }],
    "models": ["gpt-5.4", "claude-sonnet-5", "gemini-3.7-flash"],
    "rank_by": "balanced",
    "max_tokens": 512
  }'

Request

Parameter	Type	Default	Description
`messages`	array	required	1–20 `system`, `user`, or `assistant` messages; string content up to 50,000 chars.
`models`	array	server default set	2–8 concrete chat model IDs, each at most once. `ninja/auto` and other virtual IDs are rejected; a duplicate returns `400 validation_error`. Omit it to compare a built-in default set.
`rank_by`	string	`balanced`	`quality`, `speed`, `cost`, or `balanced`.
`max_tokens`	integer	`1024`	Output ceiling per model, 1–8,192. Also sizes the hold.
`temperature`	number	`0.7`	0–2, applied to every model.
`include_full_responses`	boolean	`true`	`false` truncates each `content` to 200 characters.
`stream`	boolean	`false`	Stream per-model results over SSE.

Send an Idempotency-Key so a retry replays instead of re-running every model. A streamed comparison can’t be replayed — reusing its key returns 409 stream_not_replayable.

Response

{
  "request_id": "req_...",
  "winner": { "model": "gemini-3.7-flash", "name": "Gemini 3.7 Flash", "reason": "Best balance of quality (0.91), speed (1180ms), and cost ($0.0009)" },
  "results": [
    {
      "rank": 1,
      "model": "gemini-3.7-flash",
      "content": "...",
      "error": null,
      "quality": { "confidence": 0.91, "flags": [], "suggested_retry": false },
      "latency_ms": 1180,
      "cost_cents": 0.09,
      "tokens": { "prompt": 140, "completion": 96, "total": 236 },
      "success": true
    }
  ],
  "failed": [],
  "summary": {
    "fastest": { "model": "gemini-3.7-flash", "latency_ms": 1180 },
    "highest_quality": { "model": "claude-sonnet-5", "confidence": 0.94 },
    "cheapest": { "model": "gemini-3.7-flash", "cost_cents": 0.09 },
    "best_value": { "model": "gemini-3.7-flash" }
  },
  "ranked_by": "balanced",
  "models_compared": 3,
  "succeeded": 3,
  "total_cost_cents": 0.41,
  "total_cost": "$0.0041",
  "balance": "$24.95",
  "compared_at": "2026-09-01T18:20:00.000Z",
  "metadata": { "latency_ms": 2400 }
}

results holds the successful models in rank order; failed holds any model that errored or returned an empty completion (error: "empty_completion"), at zero cost. quality is the quality score computed for that answer. best_value is the highest quality-per-cent ratio. balance_warning appears when your balance is at or below $5. The numbers in the sample are illustrative.

How ranking works

`rank_by`	Score
`quality`	The quality score’s `confidence`
`speed`	Relative latency against the slowest model in the set
`cost`	Relative cost against the most expensive model in the set
`balanced`	50% quality + 30% speed + 20% cost

Streaming

With stream: true the response is text/event-stream. Frames are data: JSON objects whose type is start, then delta and model_done (or model_error) per model as tokens arrive, and finally rankings once every model has settled.

Billing and limits

Hold, then settle. The gateway holds the sum of each model’s metered maximum (input estimate plus max_tokens at that model’s rates) before fan-out. Each model then bills its own actual metered usage; failed or empty models cost nothing; the unused hold is refunded. A balance that can’t cover the hold returns 402 insufficient_credits with estimated_cost and balance.
Rate limits. Comparing M models consumes M units of the chat requests-per-minute and concurrency budgets and M × the prompt estimate of the tokens-per-minute budget. A 429 from this route adds units. See Rate limits.
Unpriced models. A model with no metered price returns 400 unpriced_model before anything runs.

Quality scoring

How quality.confidence and flags are computed.

Batch

Many prompts, one call.

Get Started

Chat

Orchestration

Production

Media & search

Reference

Request

Response

How ranking works

Streaming

Billing and limits

Quality scoring

Batch

​Request

​Response

​How ranking works

​Streaming

​Billing and limits

Quality scoring

Batch

Request

Response

How ranking works

Streaming

Billing and limits