Streaming

Set "stream": true to receive the reply as server-sent events, in the OpenAI chunk format.

curl -N https://api.opinference.com/v1/chat/completions \
  -H "Authorization: Bearer $OP_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-oss-120b",
    "messages": [{"role": "user", "content": "Count to five."}],
    "stream": true
  }'

The event streamCopy Anchor Link

data: {"id":"gen_...","object":"chat.completion.chunk","model":"openai/gpt-oss-120b","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}]}

data: {"id":"gen_...","object":"chat.completion.chunk","model":"openai/gpt-oss-120b","choices":[{"index":0,"delta":{"content":"1, 2"},"finish_reason":null}]}

: ping

data: {"id":"gen_...","object":"chat.completion.chunk","model":"openai/gpt-oss-120b","choices":[{"index":0,"delta":{},"finish_reason":"stop"}],"usage":{"prompt_tokens":14,"completion_tokens":9,"total_tokens":23}}

data: [DONE]

• Lines starting with : are keepalive comments, sent while the model is quiet. SSE clients ignore them.
• A reasoning model streams delta.reasoning before delta.content.
• The stream ends with data: [DONE].

Usage arrives on the last chunkCopy Anchor Link

The last chunk before [DONE] always carries usage, whether or not you send stream_options.include_usage. That option is not read. Earlier chunks can carry usage too, depending on the endpoint; the last one is the total, so keep the last value you see. Chunks that carry usage can have an empty choices array, so read choices[0] only when it exists:

usage = None
for chunk in stream:
    if chunk.choices:
        print(chunk.choices[0].delta.content or "", end="")
    if chunk.usage:
        usage = chunk.usage
print("\n", usage.total_tokens, "tokens")

When a stream failsCopy Anchor Link

A request that is refused before the stream opens (a bad key, an unknown model, no credit) gets an ordinary HTTP error with the error envelope.

Once the stream is open, a failure arrives as a final chunk with an error field, followed by [DONE]:

data: {"id":"gen_...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":null}],"usage":{"prompt_tokens":14,"completion_tokens":3,"total_tokens":17},"error":{"code":"provider_unavailable","message":"the stream ended before the generation completed","kind":"unavailable","generation_id":"gen_..."}}

data: [DONE]

Check every chunk for error. A stream that ends with [DONE] is not necessarily complete.

Failover stops at the first tokenCopy Anchor Link

Until the first piece of content is sent to you, a failing endpoint can be replaced by another endpoint serving the same model, and you see one clean stream. After the first piece of content, the request is committed: a failure ends the stream with the error chunk above, and no other endpoint's output is spliced into your reply. Keepalives and the opening role chunk are not content and do not commit the stream.

So a stream either completes, or ends with an error chunk holding the text produced so far. Whether to retry the whole request is your choice. A retry is a new, separately billed request. With x-op-metadata: enabled, the final chunk also carries op_metadata with "truncated": true on a cut stream.