Inference API reference

HTTP routes the inference endpoint serves, with request bodies, responses, and status codes.

The HTTP endpoints served by a deployed inference model. Each route lists its request, response, and errors. In every path, <inference-url> is the base URL of the deployment's inference endpoint.

Endpoints

Method	Path	Route
`POST`	`/v1/endpoints/<endpoint-name>/v1/chat/completions`	Chat completions
`POST`	`/v1/endpoints/<endpoint-name>/v1/audio/transcriptions`	Audio transcriptions
`GET`	`/v1/models`	List models
`GET`	`/v1/models/stats`	Model stats

Authentication

API keys are sent in the Authorization header as a Bearer token. A key must start with the csm_live_ prefix.

Authorization: Bearer csm_live_xxxxxxxxxxxxxxxxxxxxxxxx

The routes use two authentication modes.

Mode	Behavior	Used by
None	The route runs no authentication.	List models, Model stats
Conditional	CosmicAC enforces authentication only when the endpoint enables `require_auth_header`. Otherwise it records a valid key for usage tracking, and does not block a missing or invalid key.	Chat completions, Audio transcriptions

Chat completions

Returns a chat completion from the model at the named endpoint. Supports non-streaming and streaming (SSE) responses, and multimodal input with image, video, or audio parts.

HTTP request

POST <inference-url>/v1/endpoints/<endpoint-name>/v1/chat/completions

Path parameters

Parameter	Type	Required	Description
`endpoint-name`	string	Yes	Name of the inference endpoint that serves the model.

Request headers

Header	Type	Required	Description
`Content-Type`	string	Yes	Must be `application/json`.
`Authorization`	string	No	API key as a Bearer token, `Bearer csm_live_...`. Required only when the endpoint enables `require_auth_header`.

Request body

{
  "messages": [
    { "role": "system", "content": "You are a helpful assistant." },
    { "role": "user", "content": "Hello!" }
  ],
  "stream": false,
  "temperature": 0.7,
  "max_tokens": 256,
  "top_p": 0.95,
  "frequency_penalty": 0,
  "presence_penalty": 0
}

Field	Type	Required	Description
`messages`	array	Yes	Conversation history. Must contain at least one message.
`messages[].role`	string	Yes	Role of the message author. The route accepts any string.
`messages[].content`	string \| array	Yes	Message text as a string, or an array of typed parts for multimodal models. CosmicAC rejects any other type. An empty string is valid only when `role` is `assistant`.
`model`	string	No	The model to call. The serving agent's reported model name overrides this value.
`stream`	boolean	No	If `true`, the response streams as Server-Sent Events. Defaults to `false`.
`temperature`	number	No	Sampling temperature, `0` to `2`. Higher values make the output more random, lower values more deterministic.
`top_p`	number	No	Nucleus sampling, `0` to `1`. The model considers only the tokens in the top `top_p` probability mass.
`max_tokens`	number	No	Maximum number of tokens to generate in the completion. Minimum `1`.
`frequency_penalty`	number	No	Penalty scaled by how often a token has already appeared, `-2` to `2`. Higher values reduce repetition.
`presence_penalty`	number	No	Penalty applied to tokens that have already appeared, `-2` to `2`. Higher values encourage new topics.
`n`	number	No	Number of completions to generate. Minimum `1`.
`stream_options`	object	No	Streaming options. CosmicAC forwards this object to the model without validating its contents.
`stream_options.include_usage`	boolean	No	CosmicAC sets this field to `true` on every streaming request, overriding any value you send.
`mm_processor_kwargs`	object	No	Multimodal processor settings.
`mm_processor_kwargs.fps`	number	No	Frames per second to sample from video input.
`mm_processor_kwargs.max_frames`	number	No	Maximum number of frames to sample from video input.
`mm_processor_kwargs.max_pixels`	number	No	Maximum number of pixels per image or frame.
`stop`	string \| string[]	No	CosmicAC accepts this field but does not forward it to the model.

Each message uses one form or the other. When messages[].content is an array, each part takes these fields.

Field	Type	Required	Description
`type`	string	Yes	One of `text`, `image_url`, `video_url`, or `audio_url`. CosmicAC accepts any other value without further validation.
`text`	string	When `type` is `text`	The text content of the part.
`image_url.url`	string	When `type` is `image_url`	URL of the image.
`video_url.url`	string	When `type` is `video_url`	URL of the video.
`audio_url.url`	string	When `type` is `audio_url`	URL of the audio.

Response

Non-streaming response.

{
  "id": "chatcmpl-abc123",
  "object": "chat.completion",
  "created": 1700000000,
  "model": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Hello! How can I help you today?"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 10,
    "completion_tokens": 9,
    "total_tokens": 19
  }
}

The model server produces this body and CosmicAC returns it unchanged.

Field	Type	Description
`id`	string	Identifier for the completion.
`object`	string	Always `chat.completion`.
`created`	number	Unix timestamp in seconds.
`model`	string	Model that produced the completion.
`choices`	array	Generated completions. One entry unless `n` is set higher.
`choices[].index`	number	Position of the choice in the array.
`choices[].message`	object	The generated message.
`choices[].message.role`	string	Role of the message author, `assistant`.
`choices[].message.content`	string	Generated text.
`choices[].finish_reason`	string	Why generation stopped, such as `stop` or `length`.
`usage`	object	Token counts for the request.
`usage.prompt_tokens`	number	Tokens in the prompt.
`usage.completion_tokens`	number	Tokens in the completion.
`usage.total_tokens`	number	Sum of prompt and completion tokens.

Streaming response. When stream is true, the response streams as Server-Sent Events with Content-Type: text/event-stream. Each event is a data: line carrying one completion chunk, and the stream ends with data: [DONE].

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}]}

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]}

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

Field	Type	Description
`id`	string	Identifier for the completion, the same on every chunk.
`object`	string	Always `chat.completion.chunk`.
`choices`	array	Chunk contents.
`choices[].index`	number	Position of the choice in the array.
`choices[].delta`	object	The increment added by this chunk.
`choices[].delta.role`	string	Present on the first chunk only.
`choices[].delta.content`	string	Text fragment added by this chunk.
`choices[].finish_reason`	string \| null	`null` until the final chunk.
`usage`	object	Token counts, sent on the final chunk because CosmicAC sets `include_usage`.

Errors

Failed requests return an error object. The structure matches the OpenAI error response.

{
  "error": {
    "message": "No available inference agents for endpoint: <endpoint-name>",
    "type": "invalid_request_error",
    "param": "endpointName",
    "code": "model_not_found"
  }
}

Field	Type	Description
`error.message`	string	Human-readable description of the failure.
`error.type`	string	Error category.
`error.param`	string \| null	The request field that caused the failure, when CosmicAC can identify it. `null` otherwise.
`error.code`	string \| null	Machine-readable code. `null` when the failure carries no specific code.

The type field takes one of these values.

`error.type`	Meaning
`invalid_request_error`	The request was rejected.
`server_error`	The endpoint or the model server failed.
`error`	A streaming response that already started, then broke. Carries `message` only.

The code field takes one of these values.

`error.code`	Meaning
`invalid_value`	A field failed validation. `param` names the field.
`model_not_found`	The endpoint has no replicas available.
`null`	The failure carries no specific code.

The route returns these status codes.

Status	Meaning
`400`	The request body failed validation. For example, `messages` is empty, or `temperature` is outside `0` to `2`.
`401`	An API key is required for this endpoint but was missing or invalid.
`502`	The model server returned an invalid response.
`503`	The endpoint has no available capacity to serve the request.

A request with stream: true returns HTTP 200 as soon as the event stream opens. Failures after that point arrive as a single data: event carrying the error object, followed by data: [DONE]. A streaming request that finds no available capacity reports invalid_request_error with model_not_found. A stream that breaks after it starts sending chunks reports the type error and omits param and code.

Audio transcriptions

Returns a transcription of an audio file, supplied as a file upload or as a URL. The endpoint must serve a speech-to-text model, such as Parakeet. A request to a text-only endpoint fails with 400.

HTTP request

POST <inference-url>/v1/endpoints/<endpoint-name>/v1/audio/transcriptions

Path parameters

Parameter	Type	Required	Description
`endpoint-name`	string	Yes	Name of the inference endpoint that serves the model.

Request headers

Header	Type	Required	Description
`Content-Type`	string	Yes	`multipart/form-data` for a file upload or an `audio_url` field, or `application/json` for an `audio_url` field.
`Authorization`	string	No	API key as a Bearer token, `Bearer csm_live_...`. Required only when the endpoint enables `require_auth_header`.

Request body

The route accepts audio in one of these forms.

multipart/form-data with a file upload.
multipart/form-data with an audio_url field.
application/json with an audio_url field.

Field	Type	Required	Description
`file`	file	No	The audio file to transcribe, for a multipart upload. Either `file` or `audio_url` is required. The maximum size is the endpoint's `max_file_size_mb` setting.
`audio_url`	string	No	URL of the audio file to transcribe. Either `file` or `audio_url` is required. The format comes from the URL extension, or from the response content type when the URL has no extension.

Example application/json body with a URL.

{
  "audio_url": "https://example.com/audio.mp3"
}

Request size limits

Limit	Value	Description
Maximum upload size	`max_file_size_mb`	Set per endpoint by the Parakeet job that serves it. A larger upload fails with `400`.

Response

Success response.

{
  "text": "the transcribed audio text",
  "segments": [
    {
      "start": 0.0,
      "end": 2.5,
      "text": "the transcribed",
      "id": 0,
      "seek": 0,
      "tokens": ["the", "transcribed"],
      "word_count": 2,
      "temperature": 0.0,
      "avg_logprob": null,
      "compression_ratio": null,
      "no_speech_prob": null
    }
  ],
  "words": [
    { "word": "the", "start": 0.0, "end": 0.4 }
  ],
  "language": "en",
  "metadata": {
    "total_segments": 1,
    "total_words": 6,
    "audio_duration": 5.2
  },
  "processing_time": 1.234,
  "transcription_time": 1.234,
  "source": "file"
}

Field	Type	Description
`text`	string	The transcribed text.
`segments`	array	Timed segments of the transcription.
`segments[].start`	number	Segment start time in seconds.
`segments[].end`	number	Segment end time in seconds.
`segments[].text`	string	Text of the segment.
`segments[].id`	number	Position of the segment in the array.
`segments[].seek`	number	Placeholder for OpenAI compatibility. Always `0`.
`segments[].tokens`	array	Words in the segment. Falls back to the segment text split on whitespace when the model returns no word timings.
`segments[].word_count`	number	Number of entries in `tokens`.
`segments[].temperature`	number	Placeholder for OpenAI compatibility. Always `0.0`.
`segments[].avg_logprob`	null	Placeholder for OpenAI compatibility. Always `null`.
`segments[].compression_ratio`	null	Placeholder for OpenAI compatibility. Always `null`.
`segments[].no_speech_prob`	null	Placeholder for OpenAI compatibility. Always `null`.
`words`	array	Word-level timings. Words without timestamps are omitted.
`words[].word`	string	The word.
`words[].start`	number	Word start time in seconds.
`words[].end`	number	Word end time in seconds.
`language`	string	Always `en`.
`metadata`	object	Counts describing the transcription.
`metadata.total_segments`	number	Number of entries in `segments`.
`metadata.total_words`	number	Word count of the transcription, billed as output.
`metadata.audio_duration`	number	Audio duration in seconds, billed as input.
`processing_time`	number	Seconds spent transcribing.
`transcription_time`	number	Same value as `processing_time`, kept for backward compatibility.
`source`	string	Origin of the audio, `file` or `url`.

Errors

Not every failure on this route returns an error object. Requests that CosmicAC rejects return the error object shown below. Requests the model rejects, such as unsupported or corrupt audio, return the model's own error format instead.

{
  "error": {
    "message": "Endpoint '<endpoint-name>' does not support audio transcriptions. This endpoint's model accepts: text",
    "type": "invalid_request_error",
    "param": null,
    "code": null
  }
}

Field	Type	Description
`error.message`	string	Human-readable description of the failure.
`error.type`	string	Error category.
`error.param`	string \| null	The request field that caused the failure, when CosmicAC can identify it. `null` otherwise.
`error.code`	string \| null	Machine-readable code. `null` when the failure carries no specific code.

The type field takes one of these values.

`error.type`	Meaning
`invalid_request_error`	The request was rejected.
`server_error`	The endpoint has no capacity, or the model server failed.

The route returns these status codes.

Status	Meaning
`400`	`Content-Type` is not `multipart/form-data` or `application/json`, the endpoint's model does not support audio input, the upload exceeds `max_file_size_mb`, `audio_url` is malformed, or the audio format cannot be determined.
`401`	An API key is required for this endpoint but was missing or invalid.
`500`	The model server failed while transcribing the audio.
`503`	The endpoint has no available capacity to serve the request.

File validation

Validation	Status	Description
File size	`400`	Uploads larger than the endpoint's `max_file_size_mb` setting.
Audio format	`400`	Audio whose format is not one of `.wav`, `.mp3`, `.flac`, `.ogg`, `.m4a`, `.webm`, or `.mp4`.
URL format	`400`	An `audio_url` value that is not a valid URL.

List models

Lists the models available at the inference endpoint, with availability and modality metadata for each.

HTTP request

GET <inference-url>/v1/models

Response

The data array contains one entry per available model endpoint. The example below shows one ASR model and one vLLM model.

{
  "object": "list",
  "status": "healthy",
  "data": [
    {
      "id": "nvidia/parakeet-tdt-0.6b-v3",
      "object": "model",
      "created": 1783083042,
      "owned_by": "nvidia",
      "endpoint_name": "parakeet-prod",
      "agents_available": 1,
      "job_id": "e2a127bf-5f70-4127-8c42-b93d3978b6cc",
      "status": "healthy",
      "last_health_check": {
        "timestamp": 1783084228663,
        "success": true,
        "response_time_ms": 1142
      },
      "modalities": {
        "input": ["audio"],
        "output": ["text"],
        "pipeline_tag": "automatic-speech-recognition",
        "source": "fallback"
      }
    },
    {
      "id": "Qwen/Qwen2-VL-2B-Instruct",
      "object": "model",
      "created": 1783083126,
      "owned_by": "vllm",
      "root": "Qwen/Qwen2-VL-2B-Instruct",
      "parent": null,
      "max_model_len": 27000,
      "permission": [
        {
          "id": "modelperm-3a3b6d993f074cb59078a9823827ace5",
          "object": "model_permission",
          "created": 1783083126,
          "allow_create_engine": false,
          "allow_sampling": true,
          "allow_logprobs": true,
          "allow_search_indices": false,
          "allow_view": true,
          "allow_fine_tuning": false,
          "organization": "*",
          "group": null,
          "is_blocking": false
        }
      ],
      "endpoint_name": "qwen-2-prod",
      "agents_available": 2,
      "job_id": "69c06b04-caf5-4776-87a0-2df9ff255b36",
      "status": "healthy",
      "last_health_check": {
        "timestamp": 1783084228663,
        "success": true,
        "response_time_ms": 525
      },
      "modalities": {
        "input": ["text", "image", "video"],
        "output": ["text"],
        "pipeline_tag": "image-text-to-text",
        "source": "fallback"
      }
    }
  ]
}

Field	Type	Description
`object`	string	Always `list`.
`status`	string	Aggregate health across all endpoints. `healthy` when every model is healthy, `down` when every model is down or no models are listed, and `degraded` otherwise.
`data`	array	The list of available model endpoints.
`data[].id`	string	Model identifier reported by the model server.
`data[].object`	string	Always `model`.
`data[].created`	number	Unix timestamp in seconds.
`data[].owned_by`	string	Owner reported by the model server.
`data[].endpoint_name`	string	Endpoint that serves the model.
`data[].agents_available`	number	Number of serving instances available for this model.
`data[].job_id`	string \| null	Job that serves the endpoint. `null` when no agent reports one.
`data[].status`	string	Endpoint health, `healthy`, `degraded`, or `down`.
`data[].last_health_check`	object \| null	Most recent probe result. `null` before the first probe.
`data[].last_health_check.timestamp`	number	Unix timestamp in milliseconds.
`data[].last_health_check.success`	boolean	Whether the probe succeeded.
`data[].last_health_check.response_time_ms`	number \| null	Probe round trip in milliseconds.
`data[].modalities`	object	Input and output modality metadata.
`data[].modalities.input`	array	Accepted input modalities, such as `text`, `image`, `video`, or `audio`.
`data[].modalities.output`	array	Produced output modalities.
`data[].modalities.pipeline_tag`	string	Task tag, such as `automatic-speech-recognition`.
`data[].modalities.source`	string	Where the modality data came from, `huggingface`, `fallback`, or `heuristic`.

Each data[] entry also carries any other fields the model server reports, such as root, parent, max_model_len, and permission. CosmicAC passes them through unchanged.

Model stats

Returns performance and health statistics for deployed inference endpoints, including per-replica metrics.

HTTP request

GET <inference-url>/v1/models/stats

Query parameters

Parameter	Type	Required	Description
`period`	string	No	Lookback window for traffic, failures, and latency statistics. One of `1h`, `6h`, `24h`, `7d`, or `30d`. Defaults to `1h`.
`overwrite_cache`	boolean	No	If `true`, bypasses the cached response and recomputes the statistics.

Response

The data array contains one entry per endpoint. The example below shows one single-replica endpoint and one two-replica endpoint.

{
  "object": "list",
  "period": "1h",
  "data": [
    {
      "endpoint_name": "parakeet-prod",
      "job_id": "e2a127bf-5f70-4127-8c42-b93d3978b6cc",
      "status": "healthy",
      "success_rate": 100,
      "traffic": 11,
      "failures": 0,
      "avg_response_time_ms": 339.79,
      "timestamp": 1783079163257,
      "last_health_check": {
        "timestamp": 1783079106559,
        "success": true,
        "response_time_ms": 302
      },
      "replicas": [
        {
          "replica_id": "r-01",
          "job_id": "e2a127bf-5f70-4127-8c42-b93d3978b6cc",
          "status": "healthy",
          "success_rate": 100,
          "traffic": 11,
          "failures": 0,
          "avg_response_time_ms": 339.79,
          "timestamp": 1783079163256,
          "last_health_check": {
            "timestamp": 1783079106559,
            "success": true,
            "response_time_ms": 302
          }
        }
      ]
    },
    {
      "endpoint_name": "qwen-2-prod",
      "job_id": "69c06b04-caf5-4776-87a0-2df9ff255b36",
      "status": "healthy",
      "success_rate": 100,
      "traffic": 25,
      "failures": 0,
      "avg_response_time_ms": 233.17,
      "timestamp": 1783079163257,
      "last_health_check": {
        "timestamp": 1783079106559,
        "success": true,
        "response_time_ms": 257
      },
      "replicas": [
        {
          "replica_id": "r-01",
          "job_id": "69c06b04-caf5-4776-87a0-2df9ff255b36",
          "status": "healthy",
          "success_rate": 100,
          "traffic": 12,
          "failures": 0,
          "avg_response_time_ms": 235.94,
          "timestamp": 1783079163256,
          "last_health_check": {
            "timestamp": 1783079106559,
            "success": true,
            "response_time_ms": 257
          }
        },
        {
          "replica_id": "r-02",
          "job_id": "69c06b04-caf5-4776-87a0-2df9ff255b36",
          "status": "healthy",
          "success_rate": 100,
          "traffic": 13,
          "failures": 0,
          "avg_response_time_ms": 230.61,
          "timestamp": 1783079163257,
          "last_health_check": {
            "timestamp": 1783079106559,
            "success": true,
            "response_time_ms": 257
          }
        }
      ]
    }
  ]
}

Field	Type	Description
`object`	string	Always `list`.
`period`	string	The applied lookback window.
`data`	array	One entry per endpoint.
`data[].endpoint_name`	string	Name of the endpoint.
`data[].job_id`	string \| null	Job that serves the endpoint. `null` when no replica reports one.
`data[].status`	string	Endpoint health, `healthy`, `degraded`, or `down`.
`data[].success_rate`	number	Percentage of successful requests in the period.
`data[].traffic`	number	Request count in the period.
`data[].failures`	number	Failed request count in the period.
`data[].avg_response_time_ms`	number	Mean response time in milliseconds.
`data[].timestamp`	number	Unix timestamp in milliseconds when CosmicAC built the entry.
`data[].last_health_check`	object \| null	Most recent probe across the endpoint's replicas. `null` before the first probe.
`data[].last_health_check.timestamp`	number	Unix timestamp in milliseconds.
`data[].last_health_check.success`	boolean	Whether the probe succeeded.
`data[].last_health_check.response_time_ms`	number \| null	Probe round trip in milliseconds.
`data[].replicas`	array	Per-replica breakdown.
`data[].replicas[].replica_id`	string	Identifier of the replica.
`data[].replicas[].job_id`	string \| null	Job that serves the replica.
`data[].replicas[].status`	string	Replica health, `healthy`, `degraded`, or `down`.
`data[].replicas[].success_rate`	number	Percentage of successful requests in the period.
`data[].replicas[].traffic`	number	Request count in the period.
`data[].replicas[].failures`	number	Failed request count in the period.
`data[].replicas[].avg_response_time_ms`	number	Mean response time in milliseconds.
`data[].replicas[].timestamp`	number	Unix timestamp in milliseconds when CosmicAC built the entry.
`data[].replicas[].last_health_check`	object \| null	Most recent probe for this replica.

Errors

Failed requests return an error object. The structure matches the OpenAI error response.

{
  "error": {
    "message": "querystring/period must be equal to one of the allowed values",
    "type": "invalid_request_error",
    "param": "period",
    "code": "invalid_value"
  }
}

Field	Type	Description
`error.message`	string	Human-readable description of the failure.
`error.type`	string	Error category.
`error.param`	string \| null	The query parameter that caused the failure, when CosmicAC can identify it. `null` otherwise.
`error.code`	string \| null	Machine-readable code. `null` when the failure carries no specific code.

The route returns these status codes.

Status	Meaning
`400`	`period` is not one of the accepted values.

Inference API reference

Endpoints

Authentication

Chat completions

HTTP request

Path parameters

Request headers

Request body

Response

Errors

Audio transcriptions

HTTP request

Path parameters

Request headers

Request body

Request size limits

Response

Errors

File validation

List models

HTTP request

Response

Model stats

HTTP request

Query parameters

Response

Errors

See also

On this page