> ## Documentation Index
> Fetch the complete documentation index at: https://docs.ninjachat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Model compare

> Run one prompt against 2–8 models in parallel and get the results ranked by quality, speed, cost, or a balance of all three.

Compare models on the same prompt with `POST /api/v1/compare` or `compare.create`. The response includes each answer, ranking, latency and cost.

```bash cURL theme={null}
curl https://www.ninjachat.ai/api/v1/compare \
  -H "Authorization: Bearer $NINJACHAT_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: compare-2026-09-01-0001" \
  -d '{
    "messages": [{ "role": "user", "content": "Rewrite this release note for a non-technical audience: ..." }],
    "models": ["gpt-5.4", "claude-sonnet-5", "gemini-3.7-flash"],
    "rank_by": "balanced",
    "max_tokens": 512
  }'
```

## Request

| Parameter                | Type    | Default            | Description                                                                                                                                                                                 |
| ------------------------ | ------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `messages`               | array   | required           | 1–20 `system`, `user`, or `assistant` messages; string content up to 50,000 chars.                                                                                                          |
| `models`                 | array   | server default set | 2–8 **concrete** chat model IDs, each at most once. `ninja/auto` and other virtual IDs are rejected; a duplicate returns `400 validation_error`. Omit it to compare a built-in default set. |
| `rank_by`                | string  | `balanced`         | `quality`, `speed`, `cost`, or `balanced`.                                                                                                                                                  |
| `max_tokens`             | integer | `1024`             | Output ceiling per model, 1–8,192. Also sizes the hold.                                                                                                                                     |
| `temperature`            | number  | `0.7`              | 0–2, applied to every model.                                                                                                                                                                |
| `include_full_responses` | boolean | `true`             | `false` truncates each `content` to 200 characters.                                                                                                                                         |
| `stream`                 | boolean | `false`            | Stream per-model results over SSE.                                                                                                                                                          |

Send an `Idempotency-Key` so a retry replays instead of re-running every model. A streamed comparison can't be replayed — reusing its key returns `409 stream_not_replayable`.

## Response

```json theme={null}
{
  "request_id": "req_...",
  "winner": { "model": "gemini-3.7-flash", "name": "Gemini 3.7 Flash", "reason": "Best balance of quality (0.91), speed (1180ms), and cost ($0.0009)" },
  "results": [
    {
      "rank": 1,
      "model": "gemini-3.7-flash",
      "content": "...",
      "error": null,
      "quality": { "confidence": 0.91, "flags": [], "suggested_retry": false },
      "latency_ms": 1180,
      "cost_cents": 0.09,
      "tokens": { "prompt": 140, "completion": 96, "total": 236 },
      "success": true
    }
  ],
  "failed": [],
  "summary": {
    "fastest": { "model": "gemini-3.7-flash", "latency_ms": 1180 },
    "highest_quality": { "model": "claude-sonnet-5", "confidence": 0.94 },
    "cheapest": { "model": "gemini-3.7-flash", "cost_cents": 0.09 },
    "best_value": { "model": "gemini-3.7-flash" }
  },
  "ranked_by": "balanced",
  "models_compared": 3,
  "succeeded": 3,
  "total_cost_cents": 0.41,
  "total_cost": "$0.0041",
  "balance": "$24.95",
  "compared_at": "2026-09-01T18:20:00.000Z",
  "metadata": { "latency_ms": 2400 }
}
```

`results` holds the successful models in rank order; `failed` holds any model that errored or returned an empty completion (`error: "empty_completion"`), at zero cost. `quality` is the [quality score](/quality-scoring) computed for that answer. `best_value` is the highest quality-per-cent ratio. `balance_warning` appears when your balance is at or below \$5. The numbers in the sample are illustrative.

### How ranking works

| `rank_by`  | Score                                                     |
| ---------- | --------------------------------------------------------- |
| `quality`  | The quality score's `confidence`                          |
| `speed`    | Relative latency against the slowest model in the set     |
| `cost`     | Relative cost against the most expensive model in the set |
| `balanced` | 50% quality + 30% speed + 20% cost                        |

### Streaming

With `stream: true` the response is `text/event-stream`. Frames are `data:` JSON objects whose `type` is `start`, then `delta` and `model_done` (or `model_error`) per model as tokens arrive, and finally `rankings` once every model has settled.

## Billing and limits

* **Hold, then settle.** The gateway holds the sum of each model's metered maximum (input estimate plus `max_tokens` at that model's rates) before fan-out. Each model then bills its own actual metered usage; failed or empty models cost nothing; the unused hold is refunded. A balance that can't cover the hold returns `402 insufficient_credits` with `estimated_cost` and `balance`.
* **Rate limits.** Comparing M models consumes M units of the chat requests-per-minute and concurrency budgets and M × the prompt estimate of the tokens-per-minute budget. A `429` from this route adds `units`. See [Rate limits](/rate-limits).
* **Unpriced models.** A model with no metered price returns `400 unpriced_model` before anything runs.

<CardGroup cols={2}>
  <Card title="Quality scoring" icon="shield-check" href="/quality-scoring">
    How `quality.confidence` and `flags` are computed.
  </Card>

  <Card title="Batch" icon="layer-group" href="/batch">
    Many prompts, one call.
  </Card>
</CardGroup>