Limits
Request rate and concurrency limits, request size, output length and timeouts.
Rate and concurrency
| Scope | Requests per minute | Requests at the same time |
|---|---|---|
| API key | 600, or the lower value set on the key | 200 |
| Router space | 1200 | 400 |
All keys of a space share the space limits. On top of them there are shared limits per user, per payer and for the whole service. Requests at the same time counts every running request, streamed or not.
Over a limit, the router answers 429 with a message that names the limit, for example rate limit exceeded (key requests per minute) or too many concurrent requests (key). A provider's own rate limit also comes back as 429.
The responses carry no Retry-After or X-RateLimit-* headers: retry with exponential backoff and jitter, starting from 1 second.
Request and output size
| Limit | Value |
|---|---|
| Request body | 4 MiB; over it the router answers 413 |
max_tokens | 1 to 65536; without it, 4096 |
| Context | The model's context_length in GET /api/v1/models |
| Attempts per request | 3, across all fallback models |
Timeouts
| Limit | Value |
|---|---|
| Wait for the provider to start answering | 120 seconds |
| Non-streamed request | 10 minutes |
| Streamed request | 15 minutes |
| Pause in a stream | 120 seconds |
A provider timeout is a 504. For answers that may take long, use streaming.
Other limits
- The API sends no CORS headers: call it from your server, not from a browser.
- There is no idempotency key: a repeated
POSTis a new request and is billed again. Retry only after an error that says nothing was generated, see Errors. GET /api/v1/modelsneeds no key and allows 120 requests per minute per IP address; it is cached for 60 seconds.