# Primate Vision API — full documentation corpus > Generated from https://primateintelligence.ai/docs. Spec: https://api.primateintelligence.ai/v1/openapi.json --- # Quickstart # Quickstart Primate Vision answers questions about video. Upload a video (or point us at a URL), ask a question in plain English, and get a deterministic answer with confidence and clip timestamps — no hallucinations. This page gets you from zero to a verified result in about a minute, **without creating an account**. ## 1. Get a key One command. No email, no card: ```bash doc-test id=quickstart-sandbox curl -s -X POST https://api.primateintelligence.ai/v1/sandbox ``` The response contains your `pv_test_` key plus a ready-to-analyze fixture video: ```json { "object": "sandbox_key", "api_key": "pv_test_…", "mode": "test", "expires_at": "2026-07-16T00:00:00Z", "fixture_video_id": "video_…", "fixture_prompt": "Is there a person in this video?", "expected_answer": "yes" } ``` Export it: ```bash export PRIMATE_API_KEY="pv_test_…" # from the response above ``` Test keys return **deterministic canned results** for the fixture corpus — perfect for CI. They can't spend credits or touch the GPU, and expire in 7 days. Graduating to a `pv_live_` key requires [signup](https://primateintelligence.ai/signup) (a billing gate, not an integration gate — your code doesn't change). ## 2. Create an analysis Your sandbox account comes pre-seeded with a fixture video. Ask a question about it — `Prefer: wait` holds the connection until the result is ready (up to 120s), collapsing the poll loop: ```bash doc-test id=quickstart-analyze env=VIDEO_ID curl -s -X POST https://api.primateintelligence.ai/v1/analyses \ -H "Authorization: Bearer $PRIMATE_API_KEY" \ -H "Content-Type: application/json" \ -H "Prefer: wait=60" \ -d '{"video_id": "'"$VIDEO_ID"'", "prompt": "Is there a person in this video?"}' ``` ## 3. Read the result The response is the full analysis resource: ```json { "id": "an_…", "object": "analysis", "status": "completed", "result": { "answer": "yes", "confidence": 0.93, "clips": [{ "start_s": 1, "end_s": 4, "confidence": 0.93 }], "query_type": "object" } } ``` That's the whole loop: **video → question → answer**. `result.answer` is always `yes`, `no`, or `indeterminate`; `result.clips` tells you *where* in the video. ## The same thing in TypeScript ```typescript doc-test id=quickstart-ts sdk=ts import Primate from '@primate-intelligence/sdk'; const client = new Primate(); // reads PRIMATE_API_KEY const videos = await client.videos.list(); // fixture video is pre-seeded const analysis = await client.analyses.createAndWait({ video_id: videos.data[0].id, prompt: 'Is there a person in this video?', }); console.log(analysis.result?.answer, analysis.result?.confidence); ``` ## The same thing in Python ```python doc-test id=quickstart-py sdk=py from primate_intelligence import Primate client = Primate() # reads PRIMATE_API_KEY videos = client.videos.list() # fixture video is pre-seeded analysis = client.analyses.create_and_wait( video_id=videos["data"][0]["id"], prompt="Is there a person in this video?", ) print(analysis["result"]["answer"], analysis["result"]["confidence"]) ``` ## Next steps - **[Upload your own video](/docs/guides/uploading)** — presigned upload, URL ingest, or the SDK helper - **[Write better prompts](/docs/guides/prompts)** — what's answerable, structured queries, `unassessable_components` - **[Stop polling](/docs/guides/waiting)** — `Prefer: wait` vs webhooks - **[Building with an AI agent?](/docs/agents)** — the condensed machine-readable path - **[API reference](/docs/reference)** — every endpoint, generated from the OpenAPI spec --- # Primate Intelligence Public API — Agent Guide Machine-consumable reference for AI agents integrating with the Primate Intelligence Public API (`/v1`). --- ## Quick Start ### 1. Get a sandbox key (zero-human, instant) ```http POST /v1/sandbox ``` Returns a `pv_test_` key, a `fixture_video_id`, and a `fixture_prompt`. No auth required. Provisioning is IP-capped per client (rolling 24h) — hitting the cap returns `429 sandbox_limit_exceeded` with `retry_after`. **Response includes:** - `livemode: false` — all sandbox responses are always test-mode - `upgrade` — instructions for upgrading to a live key (see below) ### 2. Upgrade to a free live key (GitHub-verified) Call `POST /v1/keys/upgrade` with your `pv_test_` key. Receives a real `pv_live_` key with a **6,000-second free credit grant**. No email, no card — but a **verified GitHub account is required** (one free grant per GitHub account, ever). **Step A — first call returns the GitHub device-flow bootstrap:** ```http POST /v1/keys/upgrade Authorization: Bearer pv_test_ ``` Returns `403 github_verification_required`. The error `details` carry everything needed: `client_id`, `device_code_url` (`https://github.com/login/device/code`), and `access_token_url` (`https://github.com/login/oauth/access_token`). **Step B — GitHub device flow (standard):** 1. `POST https://github.com/login/device/code` with `client_id` (and `Accept: application/json`) → `{device_code, user_code, verification_uri, interval}` 2. Show the user `verification_uri` + `user_code` (or open it in a browser); the user signs in to GitHub and enters the code 3. Poll `POST https://github.com/login/oauth/access_token` with `client_id`, `device_code`, `grant_type=urn:ietf:params:oauth:grant-type:device_code` every `interval` seconds until it returns `access_token` No scopes are requested — the token only proves account identity. **Step C — retry the upgrade with the token:** ```http POST /v1/keys/upgrade Authorization: Bearer pv_test_ Content-Type: application/json {"github_token": "gho_..."} ``` The token is verified with GitHub and **discarded — never stored** (only the numeric GitHub account id is retained to enforce one-grant-per-account). A GitHub account that already claimed its grant gets `409 github_account_already_used` — use the billed claim flow (step 3) instead. Response includes a `pv_live_` key and `tier: "free_grant"`. Free-grant keys: - Run real GPU inference (`livemode: true`) - Have no overage — when the grant is exhausted, analyses return `402 grant_exhausted` with a link to the claim flow for billed keys ### 3. Get a billed live key (device-code flow) ```http POST /v1/keys/request ``` Returns `{device_code, claim_url, poll_interval}`. Direct the user to `claim_url`. Poll: ```http GET /v1/keys/request/{device_code} ``` Returns `{status: "pending"}` until the user approves in the browser. On approval returns the billed `pv_live_` key ONCE (subsequent polls return `410 Gone`). --- ## The `livemode` Field **Every public API resource includes a top-level `"livemode": boolean` field** (Stripe-style). | Value | Meaning | |-------|---------| | `true` | Request used a `pv_live_` key — results are real GPU inference | | `false` | Request used a `pv_test_` key or sandbox — results are deterministic canned fixtures | **Agents SHOULD verify `livemode` matches their expectation before relaying results to users or downstream systems.** A result with `livemode: false` is a fixture/test result and must never be treated as evidence about real video content. Example analysis response: ```json { "object": "analysis", "livemode": true, "status": "completed", "result": { "answer": "yes", "confidence": 0.97, ... } } ``` --- ## Test Key Restrictions Test (`pv_test_`) keys **only analyze the fixture video** seeded by `/v1/sandbox`. Attempting to analyze any other video returns: ```json { "error": { "code": "test_key_fixture_only", "message": "Test keys can only analyze the provided fixture video. Create a live key to analyze your own uploads.", "status": 403 } } ``` This prevents fabricated results on real video content. --- ## Video Metadata After `POST /v1/videos/{id}/complete`, the API probes the uploaded S3 object and populates: | Field | Type | Description | |-------|------|-------------| | `duration_s` | number \| null | Duration in seconds | | `width` | integer \| null | Frame width in pixels | | `height` | integer \| null | Frame height in pixels | | `fps` | number \| null | Frames per second | Agents can use these fields to verify the right file arrived before submitting an analysis. Probe failures are non-fatal — fields remain `null` if probing fails. --- ## Free Re-runs (platform incidents) When an analysis **fails because of a platform incident** (GPU outage, degraded inference), the failed analysis carries **`rerun_eligible: true`**. Call `POST /v1/analyses/{id}/rerun` to create a **fresh analysis at no charge** — same video, prompt/query, model, and options; new `an_` id; `usage` stays `null` on the re-run (never billed). One free re-run per failed analysis. Analyses that fail for non-platform reasons return `409 rerun_not_eligible`. If the re-run dispatch itself fails (503), your free re-run is NOT consumed — retry later. --- ## Demo Videos (public onboarding samples) `GET /v1/demo-videos` (no auth) returns curated sample videos with **precomputed results** for example prompts — `{videos: [{id, display_name, duration_seconds, thumbnail_url, results_by_prompt}]}` where `results_by_prompt` maps prompt text → `{answer, confidence, query_type, video_url, computed_at}`. Use these to inspect real result shapes before uploading anything. Documented subset only — extra fields exist but may change. --- ## Video Playback & Result History (PRI-496) **Source playback** — `GET /v1/videos/{id}` (and the list) populates `media: {url, expires_at}` on `ready` videos: a signed playback URL for the original source video, **fresh on every read with a 1-hour TTL**. When it expires, re-fetch the video for a new one — old videos always stay playable. `media` is `null` while the video is not ready or has no stored source object. **Per-video result history** — `GET /v1/videos/{id}/analyses` lists every analysis run against a video (newest first), with the same list envelope and filters as `GET /v1/analyses` (`status`, `limit`, `starting_after`). It is the resource-nested spelling of `GET /v1/analyses?video_id={id}` — use whichever fits your client. Together with `Analysis.artifacts` (annotated result video, also sign-on-read) this gives full library semantics: list videos, replay sources, and rebuild each video's analysis history. --- ## Error Codes | Code | HTTP | Retryable | Description | |------|------|-----------|-------------| | `test_key_fixture_only` | 403 | no | Test key tried to analyze a non-fixture video. Use a live key. | | `grant_exhausted` | 402 | no | Free-grant credits depleted. Use the claim flow to get a billed key. | | `insufficient_credits` | 402 | no | Live key ran out of credits. Add credits or upgrade. | | `sandbox_limit_exceeded` | 429 | yes | Too many sandbox provisions from this IP. Retry after the window. | | `upgrade_limit_exceeded` | 429 | yes | Too many free-grant upgrades from this IP today. | | `invalid_api_key` | 401 | no | Key not found or malformed. | | `key_expired` | 401 | no | Key has passed its `expires_at`. | Full registry: `GET /v1/errors` --- ## Key Lifecycle ``` POST /v1/sandbox → pv_test_ key (livemode: false, fixture-only, 7-day TTL) │ ▼ POST /v1/keys/upgrade (auth: pv_test_ key + github_token from GitHub device flow) → pv_live_ key (livemode: true, tier: free_grant, 6,000s grant, 30-day idle expiry) (one free grant per GitHub account — duplicate → 409 github_account_already_used) │ (grant exhausted → 402 grant_exhausted) ▼ POST /v1/keys/request → GET /v1/keys/request/{code} → billed pv_live_ key (device-code claim flow — user approves in browser) ``` --- ## What the Model Supports The current model is **`Darwin-preview-1.3B`** (status: **preview**, default — see `GET /v1/models`). Preview means early access: detection quality and the set of reliably detected actions are still evolving, and results may improve or change between checkpoint updates. The API contract (request/response shapes, billing) is stable. Full capability list: [Supported actions](https://primateintelligence.ai/docs/supported-actions). Design prompts as targeted visual questions. The model natively handles: | Type | Example | `answer` | `detected_count` | `clips` | Accuracy | |------|---------|----------|-----------------|--------|---------| | **Presence** | "Is there a person?" | yes/no/indeterminate | — | ✅ when yes | Stable | | **Absence** | "Are there no dogs?" | yes/no | — | — | Stable | | **Action** | "Is someone walking?" | yes/no/indeterminate | — | ✅ when yes | Beta | | **Count — bare** | "How many people walking?" | — | ✅ integer | — | Stable | | **Count — threshold** | "More than 2 people?" | yes/no | ✅ integer | — | Stable | | **Compound** | "Person AND dog?" | yes/no | — | ✅ when yes | Stable | | **Attribute** | "Is anyone in a red jacket?" | description | — | — | Stable | | **Location** | "Where is the dog?" | description | — | — | Stable | | **State** | "Is the door open?" | description | — | — | Stable | | **Segment** | "Show me where the person is" | — | — | ✅ with masks | Stable | > **Accuracy note:** Action queries (motion/activity detection) are **beta accuracy** — results may be less reliable than presence/counting queries on short or fast-moving clips. **Not supported:** audio/sound, identity (who, not what), OCR/text in frame, subjective judgment, exhaustive count over long multi-hour footage. Open-ended prompts ("Tell me what you see") return a description but have lower accuracy — prefer a targeted question. Full guide: [Prompts & queries](https://primateintelligence.ai/docs/guides/prompts) --- ## Remote MCP Endpoint Primate Intelligence exposes a hosted [Model Context Protocol](https://modelcontextprotocol.io) server at: ``` https://api.primateintelligence.ai/mcp ``` Transport: **Streamable HTTP** (MCP spec 2025-03-26+, stateless mode). No npx or local install required — any remote agent can connect directly. ### Authentication Pass your Primate Vision API key as a Bearer token in every request: ``` Authorization: Bearer pv_live_ ``` Test keys (`pv_test_…`) also work and return deterministic fixture results. Get a free key: ```bash # Instant sandbox key (no auth required) curl -X POST https://api.primateintelligence.ai/v1/sandbox # Upgrade to a live key with a 6,000-second free credit grant # (requires a GitHub device-flow token — calling without one returns 403 # github_verification_required with the client_id + URLs to complete it) curl -X POST https://api.primateintelligence.ai/v1/keys/upgrade \ -H "Authorization: Bearer pv_test_" \ -H "Content-Type: application/json" \ -d '{"github_token": "gho_..."}' ``` Requests without a valid key receive a 401 JSON-RPC error with provisioning guidance. ### Available Tools The remote endpoint exposes the same 10 tools as the npx package: `create_video_from_url`, `create_analysis`, `validate_analysis`, `create_analysis_batch`, `get_analysis`, `wait_for_analysis`, `list_models`, `get_usage`, `get_credits`, `get_test_fixture`. - `validate_analysis` — free dry-run (`validate_only: true`): compiled query, assessability, and cost estimate before you spend credits. - `create_analysis_batch` — 2–10 prompts on one video; the first is full price, each additional is billed at 50%. - `get_credits` — balance plus the per-analysis transaction ledger (`GET /v1/credits`); prefer it over `get_usage` for auditing what each analysis cost. Every tool declares an `outputSchema` (visible in `tools/list` and on the static server card at `/.well-known/mcp/server-card.json`), generated from the same schemas that validate the public /v1 responses — and every result carries `structuredContent` conforming to it, alongside the human-readable JSON text content. **`wait_for_analysis` response envelope:** the tool returns `{ analysis, retry }` — the [Analysis resource](https://primateintelligence.ai/docs/reference#analyses) unmodified under `analysis`, plus `retry: null` when the analysis reached a terminal state, or `retry: { reason: "timeout", note }` when the wait expired (call `wait_for_analysis` or `get_analysis` again). Earlier versions merged an `_mcp_note` field into the analysis object on timeout; that field is gone. ### Client Configuration Examples **Claude Desktop / MCP config JSON:** ```json { "mcpServers": { "primate-intelligence": { "type": "streamable-http", "url": "https://api.primateintelligence.ai/mcp", "headers": { "Authorization": "Bearer pv_live_" } } } } ``` **ChatGPT app directory / custom GPT:** ``` URL: https://api.primateintelligence.ai/mcp Auth: Bearer token → your pv_live_ or pv_test_ key ``` **OpenAI Agents SDK (Python):** ```python from agents.mcp import MCPServerStreamableHttp server = MCPServerStreamableHttp( url="https://api.primateintelligence.ai/mcp", headers={"Authorization": "Bearer pv_live_"}, ) ``` ### MCP Registry The server is also listed in the MCP registry at `ai.primateintelligence/mcp` with both the npx stdio package and this remote entry. See `mcp/server.json` in the repository. --- ## Result Contract ```json { "object": "analysis", "livemode": true, "status": "completed", "result": { "answer": "yes", "confidence": 0.97, "detected_count": 3, "clips": [{"start_s": 1.2, "end_s": 3.4, "confidence": 0.94}] }, "usage": { "billed_seconds": 6, "credit_balance_after": 5994 } } ``` - `result.answer` ∈ `yes | no | indeterminate`. `indeterminate` = model couldn't commit; don't ship a decision. - `result.detected_count` — populated for count-intent queries. Integer ≥ 0. - `result.confidence` ∈ [0, 1]. Zero confidence with no positive detections returns `indeterminate`, not `no`. For count queries, `confidence` applies to the detected count itself — it is the model's confidence that `detected_count` is the correct number, not merely that something was detected. See the [Count queries](#count-queries--the-contract) section for full semantics. - `result.clips[]` — present for presence/action/compound/segment when `answer: "yes"`. Null otherwise. - `query.unassessable_components[]` — lists what couldn't be evaluated (e.g. audio). API answers what it can. - `usage.billed_seconds` — seconds charged for this analysis. - `usage.credit_balance_after` — the balance immediately after **this** analysis settled. Immutable point-in-time snapshot — it never changes as later analyses run. A `null` value (`snapshot_unavailable`) means the analysis settled before the snapshot feature shipped (migration 075) and the original balance is unknowable; treat it as missing data rather than zero. - `livemode: true` = real GPU inference. Never relay `livemode: false` results as evidence about real content. - `origin` ∈ `api | console | system` — how the analysis was created (public API, dashboard upload, or internal). **System-initiated analyses are never billed** — only `api`/`console` analyses reserve and settle credits. - `narrative` (when created with `options: {narrative: true}`): `{status: "generating"|"ready"|"failed", entries: [{t_s, text}]}` — timestamped event sentences for the video. Generation runs asynchronously **after** the analysis completes: the first completed read may show `status: "generating"` with empty entries — poll the GET until `ready`. Included in the analysis price (no surcharge). Without the opt-in: `narrative: null`, always. - `artifacts` (completed analyses with an annotated result video): `{annotated_video_url, expires_at}` — a **fresh 1-hour signed URL on every GET**; when it expires, re-fetch the analysis for a new one. `null` when no annotated video exists. --- ## Count queries — the contract A count query ("how many X…", "more than N X?") succeeds like this: ```json doc-test id=count-contract-success { "answer": "yes", "confidence": 0.94, "detected_count": 3, "clips": [{ "start_s": 1.2, "end_s": 3.4, "confidence": 0.94, "terms": { "person": 0.94 } }], "term_confidences": { "person": 0.94 }, "query_type": "object", "video_duration_s": 6.0, "indeterminate_reason": null } ``` And fails like this: ```json doc-test id=count-contract-failure { "answer": "indeterminate", "confidence": 0, "detected_count": 0, "clips": [], "term_confidences": {}, "query_type": "object", "video_duration_s": 6.0, "indeterminate_reason": "nothing_detected" } ``` **Rules:** - `detected_count` is only meaningful when `answer` is determinate (`yes` or `no`). - `detected_count: 0` with `answer: "indeterminate"` means the pipeline found **nothing assessable** — NOT "zero occurrences". - A true "zero occurrences" result is `answer: "no"`, `detected_count: 0`. - **Never branch on `detected_count` without checking `answer` first.** **Confidence semantics for count queries:** for count queries, `confidence` applies to the detected count itself — it is the model's confidence that `detected_count` is the correct number, not merely that something was detected. `{"confidence": 0.94, "detected_count": 3}` asserts "there are 3" at 94% confidence, not "there is at least one". Numerically it is derived from the per-term detection confidences underlying the count (max across detected clips, clamped to [0, 1]). When `answer` is `indeterminate`, `confidence` is always 0 (enforced by the API regardless of what inference returned); `indeterminate_reason: "nothing_detected"` additionally forces `detected_count` to 0 because no objects were seen at any confidence level. Treat `confidence ≥ 0.7` as reliable for production decisions, `0.4–0.69` as marginal (verify with a second query or tighter prompt), and `< 0.4` as unreliable. Example: `{"answer": "yes", "confidence": 0.94, "detected_count": 3}` means the model is 94% confident the count is exactly 3 — act on it; `{"answer": "indeterminate", "confidence": 0, "detected_count": 0, "indeterminate_reason": "nothing_detected"}` means no objects were detected at any confidence — do not infer "zero occurrences". --- ## Query-Type Maturity | Query type | Maturity | Notes | |---|---|---| | **presence** (`object`) | GA | "Is there X in this video?" — most reliable query form. | | **counting** (`object` + count intent) | GA | "How many X?" — returns `result.detected_count`. | | **action** / **temporal** | Beta | "Does X happen during the first N seconds?" — duration-sensitive; result reliability depends on accurate video duration metadata. High `duration_mismatch` rate if source duration is missing. | | **open-ended** | Rejected | Freeform prompts that don't resolve to a closed yes/no or count question are rejected immediately with `answer: "indeterminate"`, `indeterminate_reason: "unsupported_query_form"`, at **zero cost** — no credits billed. Rephrase as a presence or count query. | **Agent guidance:** - Prefer presence and counting queries for production pipelines. - Action/temporal queries require the source video to have accurate `duration_s` metadata (populated after `POST /v1/videos/:id/complete`). - If you receive `indeterminate_reason: "unsupported_query_form"`, rewrite the prompt before retrying — retrying the same open-ended form will always fail. The result also carries `unsupported_action: {reason, docs_url}` (PRI-516) — `docs_url` points to the [supported-actions capability list](https://primateintelligence.ai/docs/supported-actions), which documents exactly what the model can and cannot detect (including the full 532-class action vocabulary). Consult it before rewriting. --- ## Action pipeline verification A ground-truth walking fixture is available for integration testing. Use it to verify the full pipeline (upload → analyze → result) against a known answer before deploying to production. ### Walking fixture | Field | Value | |-------|-------| | **URL** | *(URL pending — clip `IMG_3281.MOV` not yet uploaded to CDN; see Slack thread for ETA)* | | **CDN key** | `fixtures/walking-ground-truth.mov` on `d3silto12vjvss.cloudfront.net` (pending upload) | | **Scene** | Outdoor walkway; several people walking. Clip is ~6–10 seconds. | | **Ground truth** | At least 2 people visibly walking during the clip. | | **Expected ideal API answer** | `POST /v1/analyses` with prompt `"Is someone walking?"` → `answer: "yes"`, `confidence ≥ 0.8`; with prompt `"How many people are walking?"` → `detected_count ≥ 2`, `confidence ≥ 0.7`. | > **Note:** this fixture is for action-query validation. For basic presence testing, use the sandbox fixture (`POST /v1/sandbox` → `fixture_video_id`). --- ## Batch analyses & discounts Run 2–10 prompts against the same video in a single request: ```http POST /v1/analyses/batch Authorization: Bearer pv_live_ { "video_id": "video_01J...", "prompts": ["Is there a person?", "How many people are walking?"] } ``` **Pricing rule:** the first prompt bills at full price; each additional prompt is discounted by `batch_discount_pct` (served by `GET /v1/credit-pricing`; currently **50%**, i.e. additional prompts bill at half price). The discount is config-driven — read it from the endpoint instead of hardcoding. Credits are reserved at the discounted rate, so the discount is observable directly in the ledger (`GET /v1/credits` will show a smaller `seconds_delta` for analysis 2+). **Worked example:** 6-second video at 1¢/s: - Prompt 1 (full price): 6s × 1¢ = **6¢** - Prompt 2 (50% off): 3s × 1¢ = **3¢** - Prompt 3 (50% off): 3s × 1¢ = **3¢** - Total for 3 prompts: **12¢** (vs 18¢ if billed separately) **Response shape:** ```json { "object": "analysis_batch", "id": "batch_...", "video_id": "video_01J...", "analyses": [ { "id": "analysis_01J...", "status": "queued", ... }, { "id": "analysis_01J...", "status": "queued", ... } ], "pricing": { "full_price_prompts": 1, "discounted_prompts": 1, "discount_pct": 50 } } ``` Each analysis in `analyses[]` can be polled individually via `GET /v1/analyses/{id}`. **Listing / filtering analyses:** `GET /v1/analyses` supports `status`, `video_id`, `model`, `created_after`, `created_before` plus cursor pagination (`limit`, `starting_after`). The `status` filter takes the public vocabulary: `queued | preparing | analyzing | rendering | completed | failed | canceled` (use `status=analyzing` for currently-running analyses; unknown values → 400 `validation_failed`). **Dry-run (`validate_only`):** pass `"validate_only": true` to parse all prompts and get per-prompt cost estimates without reserving credits or creating jobs (HTTP 200): ```json { "object": "analysis_batch_preview", "video_id": "video_01J...", "prompts": [ { "index": 0, "query": {...}, "parse_mode": "heuristic", "assessable": true, "estimated_seconds": 10, "estimated_cost_usd": 0.10, "discount_pct": 0 }, { "index": 1, "query": {...}, "parse_mode": "heuristic", "assessable": true, "estimated_seconds": 5, "estimated_cost_usd": 0.05, "discount_pct": 50 } ], "pricing": { "full_price_prompts": 1, "discounted_prompts": 1, "discount_pct": 50, "estimated_total_seconds": 15, "estimated_total_cost_usd": 0.15 } } ``` Estimates are `null` when the video has no known duration (still processing or URL-sourced before probe completes). > **Note:** sending a `prompts` array to `POST /v1/analyses` returns a validation error with a pointer to this endpoint. --- ## Streaming (real-time video over WebRTC) `GET /v1/models` advertises streaming support per model — `Darwin-preview-1.3B` carries `capabilities: {prompt, structured_query, narrative, streaming: true}`. (The former id `darwin-1.3` remains accepted on requests as a deprecated alias — responses always emit the canonical id.) Streams analyze live video in real time — same prompt semantics and same result contract as file analyses, delivered per-frame over a signaling WebSocket. ### Lifecycle ``` POST /v1/streams → queued|ready → (WS join → offer/answer/ICE) → live → ended ``` 1. `POST /v1/streams {prompt}` (secret key) → returns `signaling.url`, `ice_servers` (STUN + TURN with credentials), `limits`. 2. `POST /v1/client_tokens {scopes: ["streams:signal"], stream_id, ttl_s}` → `pvct_` token for the device. The signaling WS **never** accepts secret keys. 3. Connect `signaling.url?token=pvct_…`, send `join`, receive `ready` (or `queued {position}`), then standard WebRTC offer/answer + **bidirectional trickle ICE** (`ice {candidate}` messages flow both ways — the server trickles late-gathered srflx/relay candidates after its answer; keep consuming them). 4. `live` → `result {frame_num, detections}` per analyzed frame, `metering {elapsed_s, billed_s, session_remaining_s}` every 5s, `warning {remaining_s}` before credit exhaustion, `end {reason}`. 5. Mid-stream, send `{"type": "update_prompt", "prompt": "…"}` (client→server, ≤2000 chars) to change the question without reconnecting — the server recompiles it, applies it to the live engine, and confirms with `{"type": "prompt_updated"}`. Subsequent result frames echo the new prompt. Invalid prompts return `{"type": "error", "code": "validation_failed" | "parse_failed"}` and the old prompt stays active. On narrative-opted-in streams, expect `status` events (e.g. `recalculating`) as the engine rebuilds context after the update. ### Metering tick fields Every 5s while live: `{type: "metering", elapsed_s, billed_s, session_remaining_s, balance_s}`. - `session_remaining_s` — seconds remaining in **this session's** credit reservation (session cap), NOT your account credit balance. Account balance lives at `GET /v1/billing/credits` (`balance_seconds`). - `balance_s` — **deprecated** alias of `session_remaining_s` (identical value). Removed **~2026-08-28**; migrate reads to `session_remaining_s`. - `elapsed_s` / `billed_s` — live-clock seconds elapsed = billed (identical by construction; join/negotiation free). ### Result sampling (results are sampled, not per-frame) Results are **sampled**, not emitted for every source frame — inference cadence is adaptive (roughly every 8th source frame under load). `frame_num` is the **source-frame index** the result was computed on (so gaps between consecutive `frame_num` values are normal), and `results_summary.result_frames` on the terminal resource counts the number of **result events emitted** — the two are different axes. Expect results at roughly 8–15/s depending on load; do not assume a fixed cadence. ### No-media warning (the server tells you when YOUR media is the problem) If the WebRTC transport connects but **no decodable video frame arrives within 5 seconds**, the server pushes a `warning` event on the signaling WS (re-warned once at 15s, then quiet): ```json {"type": "warning", "code": "no_media_frames", "transport_connected": true, "elapsed_s": 5, "packets_received": 312, "frames_decoded": 0, "hint": "…"} ``` Read `packets_received` to self-diagnose: - `packets_received: 0` — your client is **not sending media** (track not attached, muted, or the capture source is dead). - `packets_received > 0` with `frames_decoded: 0` — media is arriving but is **not decodable** (wrong codec — must be VP8 or H.264 — or the source produces no real frames, e.g. a canvas capture without an active draw loop). If the session then ends without ever going live, the terminal resource records `end_reason: "media_timeout"` with the same counters in `failure_diagnostic`, and bills 0. ### Terminal results_summary The ended stream resource carries `results_summary: {result_frames, frames, last_detections}`: - `result_frames` — number of result events emitted over the session (sampled — see “Result sampling” above; this is NOT a source-frame count). - `frames` — **deprecated** alias of `result_frames` (identical value). Removed **~2026-08-28**; migrate reads to `result_frames`. - `last_detections` — the final `result.detections[]` rows, same contract as live results. ### Streaming result contract (identical to file analyses — transport never changes enums) Each `result.detections[]` row: - `answer`: `"yes" | "no" | "indeterminate"` — lowercase, same enum as file-API results - `confidence`: 0..1; `term_confidences`: per-term map - `prompt`: echoed **exactly as you submitted it** (byte-identical) - `query_type`, `search_terms`, `prompt_intent`: same vocabulary as `POST /v1/parse` - `frame_num`, `elapsed_s`, `session_id`, `session_fps` - `timing`: server-side latency telemetry (per-stage breakdowns: inference, encode, per-hop p50s). Rich, production-grade, safe to log — field names may grow, existing names are stable. - `narrative_update` (only when the stream was created with `options: {narrative: true}`): `{t_s, text}` — a new narrative sentence, emitted event-driven when the engine detects an appearance/disappearance/action, NOT per frame. Absent (not `null`) on frames without one. Terminal `results_summary.last_detections` never carries it. ### Status events (narrative-opted-in streams only, PRI-496) Streams created with `options: {narrative: true}` also receive a `status` server→client event on the signaling WS: ```json {"type": "status", "status": "prompt_context" | "combined_prompt" | "recalculating", "message": "…"} ``` Use these to order/annotate narrative entries (e.g. mark when the engine is recalculating context after a prompt update). The vocabulary is a **closed set** — exactly these three values; internal engine statuses never leak. Streams without the narrative opt-in never receive `status` events (their event vocabulary is unchanged). ### Session recordings (PRI-496) Create the stream with **`recording: true`** (top-level boolean) to retrieve the server-side session recording afterwards: 1. The stream resource carries `recording: {status}` — `recording` while live, `available` once ended with a stored recording, `failed` if capture failed, `none` if nothing was stored. Streams without the opt-in have `recording: null`. 2. Once ended with `recording.status: "available"`, call **`GET /v1/streams/{id}/recording`** → `{url, expires_at, content_type: "video/h264", container: "h264-annex-b"}`. The URL is **signed fresh on every GET with a 1-hour TTL** — re-fetch for a new one; old recordings always stay retrievable. 3. The recording is a raw H.264 Annex-B elementary stream (the exact annotated frames the client saw). Lossless remux to MP4: `ffmpeg -i recording.h264 -c copy recording.mp4`. ### end_reason vocabulary (honest by construction) | Reason | Meaning | Billed? | |---|---|---| | `completed` | Normal end after the session was live | live seconds | | `canceled` | Ended before ever going live (client action) | 0 | | `ice_failed` | WebRTC ICE never connected — see `failure_diagnostic` | 0 | | `media_timeout` | Transport connected but no media/results flowed | 0 | | `insufficient_credits` | Balance exhausted mid-stream | live seconds | | `timeout` | `limits.max_session_s` cutoff reached | live seconds | | `error` | Server-side failure | live seconds (0 if never live) | **A stream that never went live is never `completed`** — the API enforces this. On `ice_failed`, the terminal resource carries `failure_diagnostic` (`{local_candidates, hint}`) — the server-side ICE candidate summary. If `local_candidates` shows only private addresses (`172.x`, `10.x`), the fault is server-side config; if it shows srflx/relay candidates, check your client's network path to the TURN servers in `ice_servers`. ### Client profiles that MUST work (and are CI-tested) - Browser on home/office NAT (webcam) — the easy case - **Datacenter/CI/edge-fleet clients: UDP-blocked, TURN-over-TCP:443 relay-only** — the demanding case. The server advertises a public host candidate and srflx/relay candidates (trickled when gathering outlasts the answer). Regression-tested every deploy with a relay-only aiortc client. ### Stream a file (regression harness recipe) Re-running a known file through the live path is the natural streaming regression test. The examples repo ships `stream_file.py` (aiortc): loops a video file as the WebRTC source, applies your prompt, and audits the full session (candidates, states, results, billing) to JSON. See `python/streaming/` in the examples repo: **https://github.com/Primate-Intelligence/primate-examples** (public — no auth needed) ### Billing Billed per second of **live clock time** (join/queue/negotiation free), from the same credit ledger as uploads. Sessions that never go live bill exactly 0. Metering ticks arrive every 5s; `usage.billed_seconds` on the terminal resource is the reconciled charge. --- ## Pricing Billing is metered in **source-clock video-seconds** — the duration of video analyzed, independent of resolution or fps. Current rates are discoverable (public, no auth): ```http GET /v1/credit-pricing ``` Key fields: | Field | Meaning | |-------|---------| | `price_per_second_cents` | Price per billed video-second, in cents | | `signup_grant_seconds` | Free credit-seconds on signup / free-grant upgrade | | `allowed_purchase_cents` | Preset top-up amounts | | `batch_discount_pct` | Percent discount for each prompt after the first in `POST /v1/analyses/batch` | | `batch_min_prompts` / `batch_max_prompts` | Allowed prompts-per-batch range | **Cost of an analysis** = `usage.billed_seconds × price_per_second_cents`. Worked example: a 6-second analysis at 1¢/second → `6 × 1 = 6` cents = **$0.06**. Always read rates from the endpoint rather than hardcoding — pricing is config-driven and can change without an API version bump. --- ## API versioning Every response — success, error, even 404 — carries an `X-Api-Version` header: ``` X-Api-Version: + ``` Example: `X-Api-Version: 0.1.0+a7c1993`. The value is computed once at boot and is stable for the lifetime of a deploy. **Agent guidance:** if the value changes between two requests in the same session, a deploy happened mid-session. Re-check the [changelog](https://api.primateintelligence.ai/docs/changelog.md) before attributing new behaviour to a bug in your integration. --- ## Rate Limits All public endpoints respond with `X-RateLimit-*` headers. Retry after `Retry-After` on `429`/`503`. Sandbox/upgrade provisioning is additionally IP-rate-limited (3 sandbox provisions / IP / 24h; configurable global daily cap on free-grant upgrades). --- ## Changelog All API behaviour changes are recorded in [`/docs/changelog.md`](https://api.primateintelligence.ai/docs/changelog.md), sorted newest-first. Subscribe via RSS: [`/docs/changelog.xml`](https://api.primateintelligence.ai/docs/changelog.xml). This file and the changelog ship inside the API deploy artifact itself — `https://api.primateintelligence.ai/docs/agents.md` is the canonical copy, and `https://primateintelligence.ai/docs/agents.md` serves the same bytes (the website proxies the API). Spec, quickstart, and changelog therefore update atomically with the code they describe. **Agent guidance:** check the changelog when `X-Api-Version` changes between requests in the same session — a deploy happened mid-session and a new feature or fix may affect your integration. > Canonical, always-current copy: https://api.primateintelligence.ai/docs/agents.md (also served at https://primateintelligence.ai/docs/agents.md — same bytes, proxied). --- # Uploading video # Uploading video There are three paths to get video into Primate Vision. All of them end with a `video` resource in status `ready`, at which point you can create analyses against it. **Formats:** `video/mp4` (H.264) and `video/quicktime`. **Max size:** 2 GiB. **Retention:** source videos are deleted 30 days after upload (see [security model](/docs/guides/security-model)). ## Choosing a path | Path | Use when | |---|---| | **URL ingest** | The video is already at a public https URL (CDN, S3, YouTube-dl output) | | **Presigned upload** | You have the bytes (server file, browser file input) — bytes go straight to S3, never through our API servers | | **SDK `upload()` helper** | You use an SDK and want the presigned dance done for you | ## URL ingest One call. The API fetches the video asynchronously: ```bash doc-test id=uploading-url skip-live curl -s -X POST https://api.primateintelligence.ai/v1/videos \ -H "Authorization: Bearer $PRIMATE_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url": "https://example.com/clip.mp4"}' ``` The video starts `processing` and transitions to `ready` (or `failed` with `error.code` = `url_fetch_failed` / `url_forbidden` / `video_unreadable`). Poll `GET /v1/videos/{id}` or subscribe to the `video.ready` webhook. URL rules (SSRF policy): `https` only, port 443, public hosts only, max 3 redirects, 2 GiB cap. Private/internal addresses are rejected with `url_forbidden`. ## Presigned upload Three steps — create, PUT, complete: ```bash # 1. Create — declares filename/type/size, returns a presigned S3 PUT URL curl -s -X POST https://api.primateintelligence.ai/v1/videos \ -H "Authorization: Bearer $PRIMATE_API_KEY" \ -H "Content-Type: application/json" \ -d '{"filename": "clip.mp4", "content_type": "video/mp4", "size_bytes": 12345678}' # 2. PUT the bytes directly to S3 (the upload.url + upload.headers from step 1) curl -s -X PUT "$UPLOAD_URL" -H "Content-Type: video/mp4" --data-binary @clip.mp4 # 3. Tell the API the PUT finished curl -s -X POST https://api.primateintelligence.ai/v1/videos/$VIDEO_ID/complete \ -H "Authorization: Bearer $PRIMATE_API_KEY" ``` The presigned URL expires (see `upload.expires_at`, typically 1h). If the declared `size_bytes` doesn't match what landed in S3, `complete` fails with `upload_incomplete` — re-create the video. ## SDK helper ```typescript import Primate from '@primate-intelligence/sdk'; import { readFile } from 'fs/promises'; const client = new Primate(); const video = await client.videos.upload(await readFile('clip.mp4'), { filename: 'clip.mp4', content_type: 'video/mp4', }); ``` ```python from primate_intelligence import Primate client = Primate() with open("clip.mp4", "rb") as f: video = client.videos.upload(f.read(), filename="clip.mp4") ``` Both run create → PUT → complete and return the video resource. ## Browser uploads Never put a secret key in a browser. Mint an ephemeral [client token](/docs/guides/security-model#client-tokens) server-side with the `videos:write` scope, then run the presigned flow from the browser with the `pvct_` token. Working example: the [browser SPA sample](https://github.com/Primate-Intelligence/primate-examples). ## Lifecycle ``` awaiting_upload → uploading → processing → ready ↘ failed ``` `DELETE /v1/videos/{id}` removes the video (409 `resource_conflict` while analyses are still running against it). Deletion propagates from S3 + CDN within 72h; signed URLs expire within 1h regardless. ## Your video library (playback + history) Uploaded videos stay usable as a library — three fields/endpoints together give full list-replay-history semantics: - **`Video.media`** — ready videos carry `media: {url, expires_at}`, a signed playback URL for the original source video. Sign-on-read: generated **fresh on every GET with a 1-hour TTL** — re-fetch the video for a new URL after expiry; old videos always stay playable. `media` is `null` for non-ready videos and sandbox fixture videos. - **`GET /v1/videos/{id}/analyses`** — every analysis ever run against a video, newest first, with the same list envelope + filters (`status`, `limit`, `starting_after`) as `GET /v1/analyses`. (Equivalent spelling: `GET /v1/analyses?video_id={id}`.) - **`Analysis.artifacts`** — completed analyses with an annotated result video carry `annotated_video_url` (same sign-on-read semantics). --- # Prompts & queries # Prompts & queries Every analysis is a **question about a video**. You can ask it two ways: free text (`prompt`) or a pre-compiled structured query (`query`). Provide exactly one. ## What Primate can answer The model supports ten distinct prompt types. The table below shows what to expect from each. | Type | Example prompt | `answer` | `detected_count` | `clips` | |------|---------------|----------|-----------------|---------| | **Presence** | "Is there a person?" | yes / no / indeterminate | — | ✅ when yes | | **Absence** | "Are there no dogs?" | yes / no | — | — | | **Action** | "Is someone walking?" | yes / no | — | ✅ when yes | | **Count — bare** | "How many people are in this video?" | — | ✅ integer | — | | **Count — threshold** | "Are there more than 2 people?" | yes / no | ✅ integer | — | | **Compound** | "Is there a person AND a dog?" | yes / no | — | ✅ when yes | | **Attribute** | "Is anyone wearing a red jacket?" | description | — | — | | **Location** | "Where is the dog?" | description | — | — | | **State** | "Is the door open?" | description | — | — | | **Segment** | "Show me where the person is" | — | — | ✅ with masks | **Open-ended prompts** ("Tell me what you see") return a description but have lower accuracy than targeted queries. Prefer a specific question. **Not supported:** - Audio / sound ("Is there loud noise?") - Identity ("Is this John?") - OCR / text in frame ("What does the sign say?") - Subjective judgment ("Does the room look nice?") - Exhaustive counting over long footage ("Count every bird in this 2-hour archive") — count queries are accurate on a single clip or short segment, not long multi-hour recordings --- ## Free-text prompts ```json { "video_id": "video_…", "prompt": "Is there a person near the door?" } ``` The server compiles your prompt into a structured query (the `query` field on the analysis shows exactly what it understood, `parse_mode` shows how: `llm` or `heuristic`). Max 2000 characters. ### Prompt design tips **Do this:** - "Is there a person near the door?" — presence - "Are there more than two people in frame?" — count + threshold, returns `yes`/`no` + `detected_count` - "How many people are walking?" — bare count, returns `detected_count` + natural-language narrative - "Does anyone fall down?" — action - "Is there a person AND a dog?" — compound - "Is anyone wearing a red jacket?" — attribute - "Is the door open or closed?" — state - "Where is the bicycle relative to the car?" — location **Avoid this:** - "Tell me everything that happens in the video" — open-ended; use a targeted question instead - "Count every pedestrian across this 30-minute file" — count works best on clips, not long recordings - "Is this suspicious activity?" — subjective; rephrase as a specific visual condition **One question per analysis beats a paragraph.** If you need to answer "Is there a person?" AND "Are there more than 2 people?", submit two analyses — the answers are independent and the second one gives you `detected_count`. --- ## The answer contract ### `result.answer` `result.answer` is one of `yes`, `no`, `indeterminate`: | Value | Meaning | |-------|---------| | `yes` | The condition is present with meaningful confidence | | `no` | The condition was not detected | | `indeterminate` | The model could not commit either way — treat as "don't ship a decision on this" | Count queries (bare "how many?" form) do not populate `result.answer`. Use `result.detected_count` instead. ### `result.detected_count` Populated for `count`-intent analyses (when the prompt asks "how many?" or uses a threshold): ```json { "result": { "answer": "yes", "confidence": 0.94, "detected_count": 3, "clips": [...] } } ``` For count + threshold queries, `answer` tells you whether the threshold was met; `detected_count` tells you the actual number observed. ### `result.confidence` `result.confidence` ∈ [0, 1]. Meaningful confidence is ≥ 0.5. A result with `confidence: 0` and no positive term detections returns `answer: "indeterminate"`, not `"no"`. ### `result.clips[]` Clip objects carry `start_s`, `end_s`, `confidence`. Present for presence, action, compound, and segment queries when the answer is `yes`. Null or absent for description/count responses. ### `result.query_type` How the question was classified: `object`, `action`, `compound`, `attribute`, or `open_ended`. ### `narrative` (opt-in) Create the analysis with `options: {narrative: true}` and the completed analysis carries a top-level `narrative` object — timestamped event sentences describing what happened in the video: ```json {"narrative": {"status": "ready", "entries": [{"t_s": 3.2, "text": "A person enters from the left."}]}} ``` Generation runs **asynchronously after the analysis completes**: the first completed read may show `status: "generating"` with empty entries — keep polling the GET until `ready` (or `failed`). Included in the analysis price, no surcharge. Without the opt-in, `narrative` is always `null`. (Streams have a live equivalent — see [Streaming § Live narrative](/docs/guides/streaming#live-narrative-opt-in).) --- ## Unassessable components If part of your question can't be assessed, the API **answers what it can and tells you what it couldn't** — it never silently drops half your question: ```json { "prompt": "Is there a person talking loudly?", "query": { "search_terms": ["person"], "unassessable_components": ["talking loudly (audio)"] }, "result": { "answer": "yes", "confidence": 0.91 } } ``` Check `query.unassessable_components` when your prompts mix visual and non-visual language. --- ## Structured queries Skip the compiler and send the compiled form directly (`parse_mode: "client"`). Useful when you generate queries programmatically or need exact control: ```json { "video_id": "video_…", "query": { "search_terms": ["person", "dog"], "conditions": [ { "type": "presence", "target": "person", "negated": false }, { "type": "presence", "target": "dog", "negated": false } ], "connective": "and", "query_type": "compound", "prompt_intent": "presence", "response_framing": "yes_no" } } ``` Key structured query fields: | Field | Values | Description | |-------|--------|-------------| | `query_type` | `object` `action` `compound` `open_ended` | Top-level classification | | `prompt_intent` | `presence` `absence` `count` `action` `segment` | What the model optimizes for | | `response_framing` | `yes_no` `description` `count` | Shape of the result | | `count_terms` | string[] | What to count (for count intent) | | `count_threshold` | integer \| null | Threshold for threshold comparison | | `count_comparison` | `gt` `gte` `lt` `lte` `eq` \| null | How to compare against threshold | | `quantifier` | `{operator, value}` \| null | Alternative to count_threshold | If the body fails the CompiledQuery schema you get `400 query_invalid` with `param` naming the field. If the compiler can't parse a free-text prompt you get `422 parse_failed` — rephrase, or fall back to a structured query. --- ## Prompt errors | Code | Meaning | |------|---------| | `prompt_empty` | Provide `prompt` or `query` | | `prompt_too_long` | > 2000 chars | | `query_invalid` | Structured query fails the schema (`param` names the field) | | `parse_failed` | Compiler couldn't produce a query — retryable after rephrasing | --- # Waiting for results # Waiting for results Analyses run asynchronously. There are three ways to learn when one finishes; pick by context. | Strategy | Use when | |---|---| | **`Prefer: wait`** | Interactive flows, scripts, first success experience — simplest | | **Polling** | You want progress bars / queue position, or waits longer than 120s | | **Webhooks** | Production pipelines, high volume, serverless — no held connections | ## `Prefer: wait` (sync sugar) Add one header to the create and the API holds the connection until the analysis reaches a terminal state (or the wait cap): ```bash doc-test id=waiting-prefer env=VIDEO_ID curl -s -X POST https://api.primateintelligence.ai/v1/analyses \ -H "Authorization: Bearer $PRIMATE_API_KEY" \ -H "Content-Type: application/json" \ -H "Prefer: wait=60" \ -d '{"video_id": "'"$VIDEO_ID"'", "prompt": "Is there a person in this video?"}' ``` - Cap: **120 seconds**. If the analysis isn't terminal by then you get the current (running) resource back — fall through to polling. - Test-mode analyses complete in seconds, so `Prefer: wait` virtually always returns the finished result. - Under capacity brownouts `Prefer: wait` may be temporarily disabled (you get an immediate 202-style response with the running resource) — code must handle a non-terminal response either way. The SDK helpers `createAndWait()` (TS) / `create_and_wait()` (Python) use this header, then fall back to polling automatically. ## Polling ```bash curl -s https://api.primateintelligence.ai/v1/analyses/$ANALYSIS_ID \ -H "Authorization: Bearer $PRIMATE_API_KEY" ``` While running, the resource carries honest progress signals: ```json { "status": "queued", "queue_position": 3, "progress": null } ``` `queue_position` and `estimated_start_s` are accurate to within ±50% — we publish transparency rather than an analysis-latency SLO. **Backoff:** poll at 1–2s intervals for interactive flows; add jitter for fleets. Terminal states are `completed`, `failed`, `canceled` — anything else keeps polling. ## Webhooks Register once, get an HTTPS POST on every terminal transition: ```bash curl -s -X POST https://api.primateintelligence.ai/v1/webhook_endpoints \ -H "Authorization: Bearer $PRIMATE_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url": "https://yourapp.com/webhooks/primate", "events": ["analysis.completed", "analysis.failed"]}' ``` See the [webhooks guide](/docs/guides/webhooks) for verification, retries, and the #1 integration rule (the webhook is a notification, not the source of truth). ## Cancellation ```bash curl -s -X POST https://api.primateintelligence.ai/v1/analyses/$ANALYSIS_ID/cancel \ -H "Authorization: Bearer $PRIMATE_API_KEY" ``` Best-effort once the analysis is `analyzing`; a 409 `analysis_not_cancelable` means it already reached a terminal state. You are not billed for canceled work that never ran. --- # Webhooks # Webhooks Webhook endpoints receive an HTTPS POST whenever a subscribed event fires. Event vocabulary (closed, v1): `video.ready`, `video.failed`, `analysis.completed`, `analysis.failed`, `analysis.canceled`, `stream.ended`. ## Rule #1: the webhook is a notification, not the source of truth Payloads over 256 KiB are truncated (`data.truncated: true`). Delivery is **at-least-once with no ordering guarantee**. The reliable pattern is always: 1. Receive the event → verify the signature → dedupe on the `webhook-id` header 2. `GET` the resource by id for the authoritative state 3. Act on that Never assume `analysis.completed` arrives before (or after) your poll sees `completed`. ## Register an endpoint ```bash curl -s -X POST https://api.primateintelligence.ai/v1/webhook_endpoints \ -H "Authorization: Bearer $PRIMATE_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url": "https://yourapp.com/webhooks/primate", "events": ["analysis.completed", "analysis.failed"]}' ``` The response contains the signing secret (`whsec_…`) **exactly once**. Store it. Max 10 endpoints per account. ## Delivery contract - Body: `{"id": "evt_…", "type": "analysis.completed", "created_at": "…", "data": {…full resource DTO…}}` - Timeout **10s**; only 2xx counts as delivered (3xx = failure) - Retry schedule: `1m, 5m, 30m, 2h, 8h, 24h` with jitter; then the event dead-letters (visible in the deliveries API for 30 days) - 72h of 100% failure → the endpoint auto-disables + you get an email. Re-enable with `POST /v1/webhook_endpoints/{id}/enable` ## Verify signatures Deliveries are signed per [Standard Webhooks](https://www.standardwebhooks.com). Three headers: `webhook-id`, `webhook-timestamp`, `webhook-signature` (`v1,`; multiple space-separated during rotation). Signed content is `` `${id}.${timestamp}.${rawBody}` ``; the HMAC key is the base64-decoded part of your `whsec_` secret. Reject skew > 5 minutes; compare constant-time. Both SDKs ship verifiers: ```typescript import { verifyWebhookRequest } from '@primate-intelligence/sdk'; app.post('/webhooks/primate', (req, reply) => { verifyWebhookRequest({ secret: process.env.PRIMATE_WEBHOOK_SECRET!, headers: req.headers, rawBody: req.rawBody, // MUST be the raw bytes, not re-serialized JSON }); // throws on failure const event = JSON.parse(req.rawBody); if (alreadyProcessed(event.id)) return reply.code(204).send(); // dedupe on evt_ id // … fetch the resource, act … reply.code(204).send(); }); ``` ```python from primate_intelligence import verify_webhook_request @app.post("/webhooks/primate") async def receive(request: Request): raw = await request.body() verify_webhook_request( secret=os.environ["PRIMATE_WEBHOOK_SECRET"], headers=dict(request.headers), raw_body=raw, # raw bytes, not re-serialized JSON ) # raises ValueError on failure event = json.loads(raw) ... return Response(status_code=204) ``` > Per-request `webhook` overrides on `POST /v1/analyses` / `POST /v1/streams` have no secret exchange and are delivered **unsigned** — register an endpoint when you need verification. ## Rotate secrets ```bash curl -s -X POST https://api.primateintelligence.ai/v1/webhook_endpoints/$ID/rotate_secret \ -H "Authorization: Bearer $PRIMATE_API_KEY" ``` Both old and new secrets sign deliveries for **24h** (the signature header carries two entries); verification passes when any signature matches *your* secret. Full key-compromise procedure: [security model](/docs/guides/security-model). ## Debug deliveries ```bash # last 100 deliveries with response codes curl -s https://api.primateintelligence.ai/v1/webhook_endpoints/$ID/deliveries \ -H "Authorization: Bearer $PRIMATE_API_KEY" # redeliver a failed/dead one now curl -s -X POST https://api.primateintelligence.ai/v1/webhook_endpoints/$ID/deliveries/$DELIVERY_ID/redeliver \ -H "Authorization: Bearer $PRIMATE_API_KEY" ``` ## Local development Use a tunnel (ngrok, cloudflared) so the API can reach your dev machine, or skip webhooks locally and use `Prefer: wait` — the two strategies are drop-in replacements for the completion signal. --- # Errors & retries # Errors & retries Every non-2xx response uses one envelope: ```json { "error": { "code": "insufficient_credits", "message": "Insufficient credits. Add credits or enable auto-refill.", "status": 402, "param": null, "docs_url": "https://primateintelligence.ai/docs/errors#insufficient_credits", "request_id": "req_01H…" } } ``` - **`code`** — stable, machine-readable, from a closed registry. Codes are append-only; meanings never change. - **`docs_url`** — deep link to the [error registry page](/docs/errors) anchor for that code. - **`request_id`** — include it in support requests; it indexes our logs. - **`error.errors[]`** — on `validation_failed`, every violation is listed (param + message), not just the first. - **`details`** — structured extras (e.g. `details.meter` on `quota_exceeded`). ## The registry is machine-readable ```bash doc-test id=errors-registry curl -s https://api.primateintelligence.ai/v1/errors ``` Each entry carries two retry flags: - **`retryable`** — safe to retry the same request unchanged (with backoff) - **`idem`** — idempotent to retry with the same `Idempotency-Key` The SDKs generate their retry policy from this table — if you build your own client, do the same and your retry logic can never drift from the API. ## What to retry | Situation | Codes | Strategy | |---|---|---| | Rate limited | `rate_limit_exceeded`, `concurrency_limit_exceeded` | Wait `Retry-After` seconds, retry | | Platform transient | `inference_unavailable`, `db_unavailable`, `internal_error`, `capacity_exhausted`, `idempotency_unavailable` | Exponential backoff + jitter, same `Idempotency-Key` | | URL fetch flake | `url_fetch_failed` | Retry the create; check the source URL is reachable | | Prompt didn't compile | `parse_failed` (422) | Rephrase the prompt or send a structured `query` | | Everything else | `validation_failed`, `invalid_api_key`, `insufficient_credits`, … | **Do not retry unchanged** — fix the cause | ## Idempotency (never double-create) Send `Idempotency-Key` (any string, 1–255 chars; UUIDv4 recommended) on `POST /v1/videos`, `/v1/analyses`, `/v1/webhook_endpoints`: - Same key + same body within 24h → the stored response replays with `Idempotency-Replayed: true` - Same key + **different** body → `409 idempotency_key_reused` - Body comparison is canonical (JCS) — key order / whitespace / number formatting never false-positive The SDKs auto-generate keys on create calls, which makes network-level retries safe by default. ## Resource-level errors Two codes never appear as HTTP errors — only inside a failed resource's `error` field: - **`inference_error`** — the model failed on this input; safe to create a new analysis - **`stuck_timeout`** — the platform lost the job; not billed; resubmit Check `analysis.error` / `video.error` when `status` is `failed`. ### Free re-runs after platform incidents Failed analyses carry **`rerun_eligible: boolean`** (present only on `failed` status) — `true` when the failure occurred during a declared platform incident or was flagged by ops. When eligible, **`POST /v1/analyses/{id}/rerun`** creates a **fresh analysis at no charge** — same video, prompt, model, and options; new id; `usage` stays `null` (never billed). One free re-run per failed analysis; calling on an ineligible analysis returns `409 rerun_not_eligible`. Check `rerun_eligible` before writing your own retry — a free re-run beats paying for a resubmit. ## A worked retry loop (Python, no SDK) ```python import time, requests def create_analysis(body, key, attempts=4): idem = str(uuid.uuid4()) for attempt in range(attempts): r = requests.post( "https://api.primateintelligence.ai/v1/analyses", json=body, headers={"Authorization": f"Bearer {key}", "Idempotency-Key": idem}, ) if r.ok: return r.json() err = r.json()["error"] registry = requests.get("https://api.primateintelligence.ai/v1/errors").json() flags = next(e for e in registry["data"] if e["code"] == err["code"]) if not flags["retryable"] or attempt == attempts - 1: raise RuntimeError(f"{err['code']}: {err['message']} ({err['request_id']})") time.sleep(int(r.headers.get("Retry-After", 2 ** attempt))) ``` (Or just use the SDK — this is exactly what it does, with the registry vendored at build time.) ## See also - [Error registry](/docs/errors) — every code, one anchor each (the `docs_url` targets) - [Rate limits & quotas](/docs/guides/rate-limits) - [Billing & credits](/docs/guides/billing) — handling `insufficient_credits` gracefully --- # Rate limits & quotas # Rate limits & quotas ## Rate limits Applied per API key, sliding window: | Class | Limit | Covers | |---|---|---| | `reads` | 600/min | All GETs | | `writes` | 300/min | POST/DELETE except analysis creation | | `analyses.create` | 60/min | `POST /v1/analyses` | Plan multipliers can raise these. Every response — including 429s — carries: ``` X-RateLimit-Limit: 600 X-RateLimit-Remaining: 597 X-RateLimit-Reset: 1751980860 ``` A 429 adds `Retry-After` (integer seconds, ≥ 1). **Honor it** — the SDKs do automatically. ```json { "error": { "code": "rate_limit_exceeded", "status": 429, "message": "Rate limit exceeded. Back off per Retry-After." } } ``` ## Concurrency limits Separate from request rates: your plan caps **simultaneous running analyses**. Exceeding it returns `429 concurrency_limit_exceeded` — wait for an active analysis to finish rather than backing off blindly. ## Quotas (metered limits) Quotas are about *how much* you process, not how fast you call. Exceeding a metered limit returns `429 quota_exceeded` with `details.meter` naming the meter. Check consumption any time: ```bash doc-test id=rate-limits-usage curl -s https://api.primateintelligence.ai/v1/usage \ -H "Authorization: Bearer $PRIMATE_API_KEY" ``` ```json { "meters": [ { "meter": "credit_seconds", "unit": "seconds", "balance": 5990 }, { "meter": "seconds_processed.period", "unit": "seconds", "used": 10, "resets_at": "2026-08-01T00:00:00Z" } ] } ``` Running out of credits is `402 insufficient_credits` (not 429) — see [billing & credits](/docs/guides/billing). ## Capacity Under platform-wide pressure new creates may get `503 capacity_exhausted` with `Retry-After` while in-flight work drains. Test-mode traffic is never shed (it doesn't touch the GPU). See [service expectations](/docs/guides/versioning#service-expectations). ## Client checklist 1. Honor `Retry-After` on every 429/503 — never hammer 2. Watch `X-RateLimit-Remaining` and pre-throttle when it gets low 3. Treat `concurrency_limit_exceeded` as "wait for completion", not "back off" 4. Treat `quota_exceeded`/`insufficient_credits` as account states — alert a human, don't retry 5. Single region (us-west-2) in v1 — no cross-region limit ambiguity --- # Test mode # Test mode Every account has two key modes. They hit the same API surface with the same code — only the backend differs. | | `pv_test_` | `pv_live_` | |---|---|---| | Getting one | `POST /v1/sandbox` — instant, anonymous | Signup + card | | Results | Deterministic, canned, fixture corpus | Real GPU inference | | Credits | Can never hold or spend credits | Metered | | Speed | Analyses complete in seconds | Workload-shaped | | Expiry | 7 days (sandbox) | Until revoked | ## Instant sandbox ```bash doc-test id=test-mode-sandbox curl -s -X POST https://api.primateintelligence.ai/v1/sandbox ``` No email, no card, no human. The response includes the key, a pre-seeded fixture video id, the fixture prompt, and the expected answer. IP-limited (default 3 provisions per 24h → `429 sandbox_limit_exceeded`). This is the zero-touch path for agents and CI: build and verify the *entire* integration before any human signs up. Graduating to live is a **billing gate, not an integration gate** — your code doesn't change. ## Deterministic fixtures On test keys, analyses against the fixture corpus return canned results — same input, same output, every time. That makes them assertable in CI: ```bash doc-test id=test-mode-fixture curl -s https://api.primateintelligence.ai/v1/test-fixture ``` ```json { "name": "presence", "test_video_url": "https://app.primateintelligence.ai/empty-state/forklift-demo.mp4", "test_prompt": "Is there a forklift in this video?", "expected_answer": "yes", "expected_confidence_min": 0.8 } ``` ## CI verification pattern ```python # conftest-friendly integration check (runs in seconds, costs nothing) from primate_intelligence import Primate def test_primate_integration(): client = Primate() # PRIMATE_API_KEY = a pv_test_ key in CI secrets videos = client.videos.list() analysis = client.analyses.create_and_wait( video_id=videos["data"][0]["id"], prompt="Is there a person in this video?", timeout_s=60, ) assert analysis["result"]["answer"] == "yes" assert analysis["result"]["confidence"] >= 0.8 ``` Every code sample in these docs runs in CI against this exact mechanism — docs that don't run don't ship. ## Limits of test mode - Fixture corpus only — you can't upload arbitrary videos for real analysis on a test key (uploads work mechanically, but analysis results are canned) - No live streaming GPU sessions - `403 test_mode_only` marks operations unavailable in your key's mode When you need real inference on your own footage: [sign up](https://primateintelligence.ai/signup), add a card, create a `pv_live_` key, change the env var. Done. --- # Versioning & deprecation # Versioning & deprecation ## Compatibility contract The v1 API is **additive-only**. Within v1 we may: - Add new endpoints, optional request fields, response fields, enum values on *new* fields, and error codes (the registry is append-only) We will never (without a major version): - Remove or rename fields/endpoints, change field types or meanings, change error-code semantics, or tighten validation on existing fields Write clients that **ignore unknown fields** — that's the only client-side requirement for staying compatible. The gate is mechanical: every spec change runs `oasdiff breaking` in CI; a breaking diff fails the build unless it's an explicitly approved major-version migration. ## Deprecation process When something must go: 1. It's marked `deprecated` in the OpenAPI spec + the [changelog](/docs/changelog) (typed entry: `breaking | additive | fix | deprecation`) 2. A **migration guide** ships with before/after code in curl/TS/Python, the deadline, and an automated check command 3. Sunset headers appear on responses where applicable 4. Breaking removals only ever land in a new major version ## Legacy `pk_` key sunset API keys created before v1 used the `pk_` prefix. They remain valid during the launch migration so existing integrations do not break. New keys and rotations now create `pv_live_` / `pv_test_` keys only. The `pk_` compatibility window is **12 months from GA announcement**. The exact sunset date starts when Matt approves the production GA cutover and will be published here, in the changelog, and in dashboard key-management copy. Before that date, rotate each legacy key from the dashboard or `POST /v1/api-keys/{id}/rotate`; the replacement key is drop-in for `Authorization: Bearer`. After the sunset, `pk_` keys return `401 invalid_api_key`. No paid usage is charged by the act of rotating a key. ## Model versioning Models are versioned resources (`GET /v1/models`) with lifecycle: `preview → stable → deprecated`, and a `sunset_at` date once deprecated. - Omitting `model` on create uses the current default — fine for exploration - **Production integrations should pin a model id** (e.g. `"model": "Darwin-preview-1.3B"`) so behavior changes only when you choose - `400 model_deprecated` past sunset tells you to re-pin; the changelog carries the migration entry ## SDK versioning SDKs follow semver. An SDK major bump happens only with an API major. Minor/patch releases are additive (new endpoints, fixes) and safe to auto-update within `^`. ## Service expectations Published targets (not a contractual SLA in v1): - **Availability:** 99.9%/mo control plane; ≥ 99% completion success for admitted analyses - **Latency:** p95 reads < 300ms; creates < 1.5s (excluding upload bytes); webhook dispatch p95 < 60s after terminal transition; test-mode analyses < 10s - **Analysis latency is workload-shaped** — we publish honest transparency instead: `queue_position` + `estimated_start_s` accurate to ±50% - Single region: us-west-2 (US). All processing and storage. EU is roadmap, not promise. ## Changelog & RSS Every change lands in the [changelog](/docs/changelog) with a date and type tag. Subscribe: [RSS feed](/docs/changelog.xml). Agents: the changelog is also in `llms-full.txt`. --- # Security model # Security model ## API keys - Two prefixes: `pv_live_` (real processing, metered) and `pv_test_` ([test mode](/docs/guides/test-mode)). 256-bit CSPRNG entropy. - Shown **exactly once** at creation/rotation. We store only a SHA-256 digest. - Send via `Authorization: Bearer …` header only. Keys never belong in URLs, logs, or browser code. - Env-var contract: `PRIMATE_API_KEY` — both SDKs and the MCP server read it. The MCP server never accepts keys as tool arguments (keys must not land in agent transcripts). - Revocation propagates in ≤ 5 seconds. ## Client tokens (browser auth) Secret keys must never reach a browser. For client-side flows, your server mints an **ephemeral client token**: ```bash curl -s -X POST https://api.primateintelligence.ai/v1/client_tokens \ -H "Authorization: Bearer $PRIMATE_API_KEY" \ -H "Content-Type: application/json" \ -d '{"scopes": ["videos:write", "analyses:read"], "ttl_s": 300}' ``` - Prefix `pvct_`, TTL 60–900s (default 900), **never refreshable** — mint a new one when it expires (`401 token_expired`) - Scopes (v1): `videos:write`, `analyses:read`, `analyses:write`, `streams:create`, `streams:signal` — a token's scopes must be a subset of the minting key's - Optionally bind a token to one `video_id` or `stream_id` — the token then works only for that resource - The streaming signaling WebSocket accepts **only** client tokens (or first-party session JWTs) — never secret keys Pattern: a ≤ 20-line "token mint" route on your server, browser does the rest. Working example: the token-mint server sample in the [examples](https://github.com/Primate-Intelligence/primate-examples). ## Data handling **Prompts and result metadata are retained to operate and improve the service; customer video content is never used to train models.** The full contract: | Data | Retention | |---|---| | Source videos | Deleted 30 days after upload; `DELETE /v1/videos/{id}` propagates from S3 + CDN within 72h (signed URLs expire ≤ 1h) | | Result artifacts (annotated videos) | 30 days, same deletion path | | Result JSON + analysis metadata | 13 months (billing/audit), then aggregated | | Prompts | Retained per the sentence above; org-level opt-out available on request | | Account deletion | Cascades videos/analyses/keys within 30 days; webhook endpoints immediately | - Encryption: TLS ≥ 1.2 in transit; AES-256 at rest - No customer video bytes in logs or analytics — enforced by a redaction test in CI - Region: all processing/storage in us-west-2 (US) - Audit log: key lifecycle + webhook changes retained 13 months ## URL ingest (SSRF policy) `POST /v1/videos {url}` fetches are locked down: https only, port 443, public addresses only (RFC-1918/link-local/metadata ranges rejected), resolved IPs pinned (no TOCTOU), max 3 redirects each re-validated, 2 GiB streamed cap, content validated by magic bytes. Policy violations return `url_forbidden`; network failures `url_fetch_failed`. ## If a key is compromised 1. **Rotate immediately** — create a new key, revoke the old one in the [dashboard](https://primateintelligence.ai/dashboard/api-keys) (propagation ≤ 5s) 2. **Audit** — `GET /v1/analyses?created_after=…` for activity you don't recognize 3. **Contact support** with request ids for usage-forgiveness on abusive spend We monitor per-key spend anomalies (> 5× 7-day baseline), page on them, and may auto-suspend a key pending confirmation — with an email + dashboard banner, never silently. ## Webhook signing Deliveries are signed (Standard Webhooks, HMAC-SHA256, 5-minute replay window, 24h dual-secret rotation). Details: [webhooks guide](/docs/guides/webhooks). --- # Billing & credits # Billing & credits Primate Vision bills by **video-seconds processed**, prepaid as credits. 1 credit-second = 1 second of video analyzed. Current pricing is always at `GET /v1/credit-pricing` (public, no auth) and on the [pricing page](/pricing). ## How credits flow - **Signup grant** — new accounts get free credit-seconds to evaluate with (currently 6,000s ≈ 100 minutes; expiring) - **Card grant** — adding a card grants another tranche - **Purchases** — buy credit blocks in the [dashboard](https://primateintelligence.ai/dashboard/billing); custom amounts supported - **Auto-refill** — opt in to top up automatically when the balance crosses the threshold (currently 600s) Test-mode keys never hold or spend credits — build and CI on `pv_test_` for free, forever. ## Check your balance ```bash doc-test id=billing-usage curl -s https://api.primateintelligence.ai/v1/usage \ -H "Authorization: Bearer $PRIMATE_API_KEY" ``` ```json { "meters": [ { "meter": "credit_seconds", "unit": "seconds", "balance": 5990 }, { "meter": "seconds_processed.period", "unit": "seconds", "used": 10, "resets_at": "2026-08-01T00:00:00Z" } ] } ``` ## What you're charged for - **Analyses** — the duration of video actually analyzed, debited when the analysis completes. Failed platform-side work (`inference_error`, `stuck_timeout`) is **not billed**; resubmit freely. - **Streams** — live sessions reserve credits up front and reconcile to actual connected seconds at end (`GET /v1/streams/{id}` shows the final usage). Mid-stream you get `warning` messages as the reservation runs low, then the session ends with `end_reason: "insufficient_credits"`. ## Handling `insufficient_credits` in code The one billing error every integration must handle (`402`, **not retryable**): ```typescript doc-test id=billing-handling sdk=ts offline import Primate, { PrimateError } from '@primate-intelligence/sdk'; const client = new Primate(); try { await client.analyses.create({ video_id: videoId, prompt: 'Is the loading dock clear?' }); } catch (err) { if (err instanceof PrimateError && err.code === 'insufficient_credits') { const { meters } = await client.usage.retrieve(); const balance = meters.find((m) => m.meter === 'credit_seconds')?.balance ?? 0; notifyOwner( `Primate Vision balance is ${balance}s — top up or enable auto-refill: ` + `https://primateintelligence.ai/dashboard/billing`, ); // Do NOT retry unchanged — queue the job for after refill instead. } else { throw err; } } ``` The robust pattern: 1. Catch `insufficient_credits` → **stop submitting** (it will keep failing) 2. Read `GET /v1/usage` for the real balance 3. Surface a human-actionable message with the billing URL 4. If the account has auto-refill, retry after a short delay **once** — refill is triggered by the threshold crossing 5. Alert *before* you hit zero: poll `credit_seconds` and warn below your own threshold ## Free-credit expiry Granted (free) credits expire; purchased credits don't. Expiry dates show in the dashboard. Expired grants simply vanish from `balance` — no negative surprises. ## Invoices & history Purchase history and receipts live in the [dashboard billing page](https://primateintelligence.ai/dashboard/billing). Usage aggregates by day/analysis are on the [usage page](https://primateintelligence.ai/dashboard/usage). --- # Streaming # Streaming Streams analyze live video in real time: point a camera (or any MediaStream) at the API and get per-frame detection results back over a WebRTC data channel, with the same prompt semantics as file analyses. ## Lifecycle ``` POST /v1/streams → queued|ready → (WS join → offer/answer/ICE) → live → ended ``` 1. **Create** the stream server-side with your secret key: ```bash curl -s -X POST https://api.primateintelligence.ai/v1/streams \ -H "Authorization: Bearer $PRIMATE_API_KEY" \ -H "Content-Type: application/json" \ -d '{"prompt": "Is there a person in the frame?", "options": {"narrative": true}, "recording": true}' ``` The response carries everything the client needs: `signaling.url` (a WebSocket), `ice_servers` (STUN/TURN with credentials), `limits` (`max_session_s`, `warn_at_remaining_s`), and `queue_position`/`estimated_start_s` when capacity is busy. Two optional opt-ins at create time: - **`options: {narrative: true}`** — [live narrative](#live-narrative-opt-in): result detections carry `narrative_update` sentences, and the WS delivers `status` ordering events. No surcharge. - **`recording: true`** — [session recording](#session-recordings-opt-in): retrieve the annotated session video after the stream ends. 2. **Mint a client token** for the browser/device (the signaling WS **never** accepts secret keys): ```bash curl -s -X POST https://api.primateintelligence.ai/v1/client_tokens \ -H "Authorization: Bearer $PRIMATE_API_KEY" \ -H "Content-Type: application/json" \ -d '{"scopes": ["streams:signal"], "stream_id": "'"$STREAM_ID"'", "ttl_s": 300}' ``` 3. **Connect** from the client. With the TS SDK this is one call: ```typescript import { connectStream } from '@primate-intelligence/sdk/browser'; const session = await connectStream({ stream, // the Stream resource (relay it from your server) clientToken, // pvct_… minted in step 2 mediaStream: await navigator.mediaDevices.getUserMedia({ video: true }), onResult: (r) => console.log(r.frame_num, r.detections), onNarrative: (n) => log(`${n.t_s}s: ${n.text}`), // narrative streams only onStatus: (s) => log(`engine: ${s.status}`), // narrative streams only onWarning: (w) => (w.code ? diagnose(w) : showCountdown(w.remaining_s)), onEnd: (reason) => cleanup(reason), }); // change the question mid-stream (server confirms with prompt_updated): session.updatePrompt('Is there a forklift moving?'); // done: session.end(); ``` Under the hood: WS `join` → `ready` (or `queued {position}`) → WebRTC `offer`/`answer` + ICE trickle → media flows → `result` messages per analyzed frame. ## Signaling protocol (build your own client) Messages you send: `join`, `offer {sdp}`, `ice {candidate}`, `update_prompt {prompt}`, `end`, `ping`. Messages you receive: `queued {position, estimated_start_s}`, `ready`, `answer {sdp}`, `ice {candidate}`, `live`, `result {frame_num, detections}`, `prompt_updated`, `status {…}` (narrative streams only), `metering {…}`, `warning {…}`, `end {reason}`, `error {code, message}`, `pong`. There are two kinds of `warning`: the credit-exhaustion countdown (`warning {remaining_s}`, no `code` field) and diagnostic warnings that carry a `code` field — see [No-media warning](#no-media-warning-the-server-tells-you-when-your-media-is-the-problem) below. Distinguish them by the presence of `code`. A Python/aiortc backend example (robotics-style, no browser — including a "stream a file" recipe and a relay-only datacenter/CI profile) ships in the public [examples repo](https://github.com/Primate-Intelligence/primate-examples), alongside a self-contained browser webcam example. **Trickle ICE is bidirectional**: after the `answer`, the server may keep sending `ice {candidate}` messages as late srflx/relay candidates finish gathering — keep consuming and adding them. Clients on UDP-blocked networks (datacenters, CI, strict NATs) complete via the TURN servers in `ice_servers`; TURN-over-TCP:443 is supported. ### Result contract (identical to file analyses) Each `result.detections[]` row uses the same enums and casing as the file API: `answer` is lowercase `yes|no|indeterminate`, `confidence` is 0..1, and `prompt` is echoed exactly as you submitted it. Latency telemetry (per-stage breakdowns) lives under a `timing` object. ### Update the prompt mid-stream Send `{"type": "update_prompt", "prompt": "…"}` (≤2000 chars) at any point while live — no reconnect, no new stream. The server recompiles the prompt, applies it to the live engine, and confirms with `{"type": "prompt_updated"}`; subsequent result frames echo the new prompt. An invalid prompt returns `{"type": "error", "code": "validation_failed" | "parse_failed"}` and the **old prompt stays active**. On narrative streams, expect a `recalculating` status event while the engine rebuilds context. ### Live narrative (opt-in) Create the stream with `options: {narrative: true}` and two things change (nothing changes without the opt-in): 1. Result detections carry a **`narrative_update`** object — `{t_s, text}`, a new narrative sentence — whenever the engine detects an appearance/disappearance/action. Event-driven, NOT per frame: absent (not `null`) on frames without one. 2. The WS delivers additive **`status`** events for ordering/annotating narrative entries: ```json {"type": "status", "status": "prompt_context" | "combined_prompt" | "recalculating", "message": "…"} ``` The vocabulary is a **closed set** — exactly these three values; internal engine statuses never leak. Typical use: mark when the engine is recalculating context right after an `update_prompt`. Included in the per-second price — no surcharge. ### Session recordings (opt-in) Create the stream with **`recording: true`** (top-level boolean) to retrieve the server-side session recording afterwards: 1. The stream resource carries `recording: {status}` — `recording` while live, `available` once ended with a stored recording, `failed`, or `none`. Streams without the opt-in have `recording: null`. 2. Once `available`, call `GET /v1/streams/{id}/recording` → `{url, expires_at, content_type: "video/h264", container: "h264-annex-b"}`. The URL is **signed fresh on every GET, 1-hour TTL** — re-fetch for a new one; old recordings always stay retrievable. 3. The recording is a raw H.264 Annex-B elementary stream (the exact annotated frames the client saw). Lossless remux to MP4: `ffmpeg -i recording.h264 -c copy recording.mp4`. ### Result sampling — results are sampled, not per-frame Results are **sampled**, not emitted for every source frame — inference cadence is adaptive (roughly every 8th source frame under load). Two different axes to keep straight: - `frame_num` on each live `result` is the **source-frame index** the result was computed on. Gaps between consecutive `frame_num` values are normal and expected — don't treat them as dropped results. - `results_summary.result_frames` on the ended stream resource counts the number of **result events emitted** over the session — NOT a source-frame count. Expect roughly **8–15 results per second** depending on load; never assume a fixed cadence. > **Alias note:** `results_summary.frames` is a supported alias of `results_summary.result_frames` (identical value), superseded by `result_frames`. Prefer `result_frames` in new code; existing reads of `frames` keep working. ### Metering ticks While live, a `metering` message arrives every 5 seconds: ```json {"type": "metering", "elapsed_s": 45, "billed_s": 45, "session_remaining_s": 555, "balance_s": 555} ``` - `session_remaining_s` — seconds remaining in **this session's** credit reservation (the session cap). This is NOT your account credit balance — that lives at `GET /v1/billing/credits` (`balance_seconds`). - `balance_s` — supported alias of `session_remaining_s` (identical value), superseded by `session_remaining_s`. Prefer `session_remaining_s` in new code; existing reads of `balance_s` keep working. - `elapsed_s` / `billed_s` — live-clock seconds elapsed = billed (identical by construction; join, queueing, and negotiation are free). ### No-media warning (the server tells you when your media is the problem) If the WebRTC transport connects but **no decodable video frame arrives within 5 seconds**, the server pushes a diagnostic `warning` on the signaling WS (re-warned once at 15s, then quiet; canceled by the first decoded frame): ```json {"type": "warning", "code": "no_media_frames", "transport_connected": true, "elapsed_s": 5, "packets_received": 312, "frames_decoded": 0, "hint": "…"} ``` Read `packets_received` to self-diagnose: - `packets_received: 0` — your client is **not sending media** (track not attached, muted, or the capture source is dead). - `packets_received > 0` with `frames_decoded: 0` — media is arriving but is **not decodable** (wrong codec — must be VP8 or H.264 — or the source produces no real frames, e.g. a canvas capture without an active draw loop). If the session then ends without ever going live, the terminal resource records `end_reason: "media_timeout"` with the same media counters in `failure_diagnostic`, and bills 0. ### Honest end reasons `end {reason}` and the terminal resource's `end_reason` distinguish real failures — a session that never went live is never `completed`: | Reason | Meaning | Billed | |---|---|---| | `completed` | Normal end after live | live seconds | | `canceled` | Ended before going live (client action) | 0 | | `ice_failed` | ICE never connected — resource carries `failure_diagnostic` (server candidate summary) | 0 | | `media_timeout` | Transport connected but no media flowed — `failure_diagnostic` carries the media counters | 0 | | `insufficient_credits` | Balance exhausted mid-stream | live seconds | | `timeout` | `limits.max_session_s` reached | live seconds | | `error` | Server-side failure | live seconds (0 if never live) | ### failure_diagnostic — debugging a failed session from the API alone When a session ends without ever going live on a transport failure, the terminal Stream resource carries a nullable `failure_diagnostic` object (and the session bills **0**): - On `ice_failed` it holds the server-side ICE candidate summary: ```json {"end_reason": "ice_failed", "failure_diagnostic": { "local_candidates": ["host 203.0.113.7", "srflx 203.0.113.7", "relay 198.51.100.20"], "hint": "Server advertised public host + srflx/relay candidates; check the client's path to the TURN servers in ice_servers." }} ``` If `local_candidates` shows only private addresses (`172.x`, `10.x`), the fault is on our side — contact support. If it shows srflx/relay candidates, check your client's network path to the TURN servers in `ice_servers` (TURN-over-TCP:443 and `turns:` TLS relays are available for locked-down egress networks). - On `media_timeout` it holds the media counters from the no-media warning (`packets_received`, `frames_decoded`) so you can tell "never sent media" apart from "sent undecodable media" after the fact. `failure_diagnostic` is `null` on successful sessions and on ends unrelated to transport. ## Credits & limits - One active (queued/ready/live) stream per account by default — a second create returns `409 stream_already_active` - Sessions reserve credits up front; metering ticks every 5s (see [Metering ticks](#metering-ticks) for the field contract); you get `warning {remaining_s}` before funds run out, then `end {reason: "insufficient_credits"}` - `limits.max_session_s` is the hard per-session cap from your plan - Your account-wide balance is at `GET /v1/billing/credits` — metering ticks only report the session reservation - After `ended`, `GET /v1/streams/{id}` shows `duration_s`, `usage` (reconciled to the second), and `results_summary` (`result_frames` = count of result events — see [Result sampling](#result-sampling--results-are-sampled-not-per-frame)) ## End a stream From the client: send `end` (or just `session.end()`). Server-side / cleanup: ```bash curl -s -X POST https://api.primateintelligence.ai/v1/streams/$STREAM_ID/end \ -H "Authorization: Bearer $PRIMATE_API_KEY" ``` Idempotent — safe to call on an already-ended stream. `stream.ended` fires as a [webhook](/docs/guides/webhooks) if subscribed. --- # Tutorial: build from scratch # Tutorial: build from scratch We'll build `askvideo` — a Node CLI that takes a video URL and a question, and prints the answer. Zero to working in about 15 minutes, entirely on a free test key. ## Step 0: get a key (30 seconds) ```bash doc-test id=tutorial-sandbox curl -s -X POST https://api.primateintelligence.ai/v1/sandbox ``` ```bash export PRIMATE_API_KEY="pv_test_…" # from the response ``` ## Step 1: project setup ```bash mkdir askvideo && cd askvideo npm init -y ``` ## Step 2: the whole program `askvideo.mjs`: ```javascript const [url, ...promptParts] = process.argv.slice(2); const prompt = promptParts.join(' '); if (!url || !prompt) { console.error('usage: node askvideo.mjs '); process.exit(2); } const apiKey = process.env.PRIMATE_API_KEY; const baseUrl = process.env.PRIMATE_BASE_URL ?? 'https://api.primateintelligence.ai'; if (!apiKey) { console.error('PRIMATE_API_KEY is required'); process.exit(2); } async function api(path, options = {}) { const res = await fetch(`${baseUrl}${path}`, { ...options, headers: { Authorization: `Bearer ${apiKey}`, 'Content-Type': 'application/json', ...(options.headers ?? {}), }, }); const body = await res.json(); if (!res.ok) { const err = body.error ?? { code: 'request_failed', message: res.statusText }; console.error(`${err.code}: ${err.message}`); if (err.docs_url) console.error(`docs: ${err.docs_url}`); process.exit(1); } return body; } async function createAndWait(videoId, question) { let analysis = await api('/v1/analyses', { method: 'POST', headers: { Prefer: 'wait=60' }, body: JSON.stringify({ video_id: videoId, prompt: question }), }); for (let i = 0; analysis.status !== 'completed' && i < 30; i += 1) { if (analysis.status === 'failed' || analysis.status === 'canceled') { throw new Error(`analysis_${analysis.status}: ${JSON.stringify(analysis.error)}`); } await new Promise(resolve => setTimeout(resolve, 1000)); analysis = await api(`/v1/analyses/${analysis.id}`); } return analysis; } const video = await api('/v1/videos', { method: 'POST', body: JSON.stringify({ url }), }); console.error(`video ${video.id} (${video.status})`); const analysis = await createAndWait(video.id, prompt); console.log(JSON.stringify(analysis.result, null, 2)); ``` ## Step 3: run it Test keys analyze the fixture corpus deterministically, so use the official fixture: ```bash FIXTURE=$(curl -s https://api.primateintelligence.ai/v1/test-fixture) node askvideo.mjs \ "$(echo $FIXTURE | python3 -c 'import json,sys; print(json.load(sys.stdin)["test_video_url"])')" \ "Is there a person in this video?" ``` ```json { "answer": "yes", "confidence": 0.93, "clips": [{ "start_s": 1, "end_s": 4, "confidence": 0.93 }], "query_type": "object", "video_duration_s": 10 } ``` That's a working integration. Everything below is production hardening. ## Step 4: make it CI-verifiable `test.mjs`: ```javascript const baseUrl = process.env.PRIMATE_BASE_URL ?? 'https://api.primateintelligence.ai'; const headers = { Authorization: `Bearer ${process.env.PRIMATE_API_KEY}`, 'Content-Type': 'application/json', }; const fixture = await fetch(`${baseUrl}/v1/test-fixture`).then(r => r.json()); const video = await fetch(`${baseUrl}/v1/videos`, { method: 'POST', headers, body: JSON.stringify({ url: fixture.test_video_url }), }).then(r => r.json()); const analysis = await fetch(`${baseUrl}/v1/analyses`, { method: 'POST', headers: { ...headers, Prefer: 'wait=60' }, body: JSON.stringify({ video_id: video.id, prompt: fixture.test_prompt }), }).then(r => r.json()); const ok = analysis.result.answer.toLowerCase() === fixture.expected_answer.toLowerCase() && analysis.result.confidence >= fixture.expected_confidence_min; console.log(ok ? 'INTEGRATION OK' : 'INTEGRATION BROKEN'); process.exit(ok ? 0 : 1); ``` Run that in CI with a `pv_test_` key in your secrets — it completes in seconds and costs nothing. ## Step 5: production hardening checklist - **Error handling** — retry only codes marked retryable in the [error registry](/docs/guides/errors-retries); never retry [`insufficient_credits`](/docs/guides/billing) - **Stop polling** — swap `createAndWait` for [webhooks](/docs/guides/webhooks) at volume - **Pin a model** — pass `model: 'Darwin-preview-1.3B'` so behavior changes only when you choose ([versioning](/docs/guides/versioning)) - **Your own videos** — swap URL ingest for [presigned upload](/docs/guides/uploading) when the bytes live with you - **Go live** — [sign up](https://primateintelligence.ai/signup), create a `pv_live_` key, change the env var. No code changes. ## Where to next - [Sample matrix](https://github.com/Primate-Intelligence/primate-examples) — Python, webhook receivers, browser SPA, streaming, token-mint server - [Agent quickstart](/docs/agents) — the condensed machine-readable version of everything you just did - [API reference](/docs/reference) --- # Supported actions # Supported actions This page is the honest capability list for **Darwin-preview-1.3B** (status: preview). It documents what the model can reliably detect, what it detects with reduced accuracy, and what it cannot detect at all — so you can design prompts that work instead of discovering the limits by trial and error. **Preview means early access**: detection quality and the set of reliably detected actions are still evolving, and results may improve or change between checkpoint updates. The API contract (request/response shapes, billing) is stable. If an analysis returns `answer: "indeterminate"` with `indeterminate_reason: "unsupported_query_form"`, or an action prompt produces a nonsensical match, this page explains why — and what to ask instead. ## Summary: what works, by prompt type | Prompt type | Example | Reliability | |---|---|---| | **Presence** | "Is there a person?" | Stable — most reliable query form | | **Absence** | "Are there no dogs?" | Stable | | **Count** (bare + threshold) | "How many people?" / "More than 2 people?" | Stable on clips; not long multi-hour footage | | **Compound** | "Is there a person AND a dog?" | Stable | | **Attribute** | "Is anyone wearing a red jacket?" | Stable | | **Location** | "Where is the dog?" | Stable | | **State** | "Is the door open?" | Stable | | **Segment** | "Show me where the person is" | Stable (returns masks) | | **Action** | "Is someone walking?" | **Beta** — see the vocabulary below | | **Open-ended** | "Tell me what you see" | Rejected at zero cost (`unsupported_query_form`) | Presence, counting, attribute, location, state, and segmentation queries recognize the visual world broadly — common objects, people, animals, vehicles, scenes. They are not limited to a fixed list. **Action queries are different**: they run against a fixed action vocabulary, and that is where the honest limits live. ## How action detection works — and why the vocabulary matters When you ask an action question ("Is someone running?", "Does anyone fall down?"), the model matches what it sees against a **fixed vocabulary of 532 action classes** (Kinetics-derived). Two consequences: 1. **In-vocabulary actions** can be detected, at beta accuracy. Short clips, fast motion, and unusual camera angles reduce reliability. 2. **Out-of-vocabulary actions cannot be detected.** The model has no embedding for the action, so it falls back to the nearest in-vocabulary neighbour — which can produce confidently wrong or nonsensical matches. If the action you care about is not in the list below, do not ship a decision on the result. When the system can tell at submit time that a requested action is outside the vocabulary, the API flags it (see [Unsupported action signal](#unsupported-action-signal) below). When it cannot tell, you may still get a low-quality nearest-neighbour result — treat unexpected action answers with suspicion and check this list. ## Known gaps (confirmed, tracked) These are actions users commonly ask about that the current vocabulary does **not** cover: - **Hand gestures** — thumbs up, thumbs down, OK sign, peace sign. (Exception: `waving hand`, `finger snapping`, `clapping`-family classes exist.) Gesture prompts currently produce unreliable nearest-neighbour matches. - **Most vehicle actions** — the vocabulary contains `driving car`, `driving tractor`, `motorcycling`, `riding scooter`, `riding a bike`, `riding mountain bike`, and `pushing car`. Essentially everything else is missing: parking, reversing, lane changes, braking, turning, overtaking, collisions/crashes, honking, loading/unloading, towing. Treat vehicle-action prompts (beyond basic "driving") as **not detected**. - **Everyday ambient actions phrased generically** — "walking" and "running" as bare actions are matched via the model's broader action understanding but the vocabulary's walking/running entries are context-specific (`walking the dog`, `running on treadmill`, `jogging`). Generic phrasings usually work but are beta-accuracy; be specific where you can. Vocabulary expansion (a ~5,000-action taxonomy with synonym grouping) is on the roadmap; this page tracks the current shipped reality. ## What is never supported (any prompt type) - **Audio / sound** — "Is there loud noise?", "What did they say?" - **Identity** — "Is this John?" (who, not what) - **OCR / reading text in frame** — "What does the sign say?" - **Subjective judgment** — "Does this look dangerous?", "Is the room clean?" (subjective terms are surfaced in `query.unassessable_components`) - **Exhaustive counting over long footage** — counts are accurate on a clip or short segment, not a multi-hour archive ## Unsupported action signal When prompt compilation determines the request cannot be scored, the API responds honestly instead of burning credits: - **Open-ended prompts** are rejected at submit time, at zero cost: the analysis completes immediately with `answer: "indeterminate"`, `indeterminate_reason: "unsupported_query_form"`, and `confidence: 0`. - Analyses whose result cannot be trusted carry an `indeterminate_reason` (`low_confidence`, `nothing_detected`, `unsupported_query_form`, `duration_mismatch`) — [full semantics in the prompts guide](/docs/guides/prompts). - Responses in the unsupported path link back to this page so agents and users can see the capability list without guessing. ## The full action vocabulary (532 classes) The complete, current list. If an action question maps to one of these, it is detectable (beta accuracy). If it does not, rephrase — often a **presence** question ("Is there a person on a ladder?") is more reliable than an action question ("Is someone climbing?"). **A** — `abseiling` · `acting in play` · `adjusting glasses` · `air drumming` · `answering questions` · `applauding` · `applying cream` · `archery` · `arm wrestling` · `arranging flowers` · `assembling computer` · `auctioning` **B** — `baby waking up` · `baking cookies` · `balloon blowing` · `bandaging` · `barbequing` · `bartending` · `base jumping` · `bathing dog` · `beatboxing` · `bee keeping` · `belly dancing` · `bench pressing` · `bending back` · `bending metal` · `biking through mud` · `blasting sand` · `blowing glass` · `blowing leaves` · `blowing nose` · `blowing out candles` · `bobsledding` · `bookbinding` · `bouncing on trampoline` · `bowling` · `braiding hair` · `breading or breadcrumbing` · `breakdancing` · `brush painting` · `brushing hair` · `brushing teeth` · `building cabinet` · `building shed` · `bungee jumping` · `busking` **C** — `canoeing or kayaking` · `capoeira` · `carrying baby` · `cartwheeling` · `carving pumpkin` · `catching fish` · `catching or throwing baseball` · `catching or throwing frisbee` · `catching or throwing softball` · `celebrating` · `changing oil` · `changing wheel` · `checking tires` · `cheerleading` · `chopping wood` · `clapping` · `clay pottery making` · `clean and jerk` · `cleaning floor` · `cleaning gutters` · `cleaning pool` · `cleaning shoes` · `cleaning toilet` · `cleaning windows` · `climbing a rope` · `climbing ladder` · `climbing tree` · `closing something` · `contact juggling` · `cooking chicken` · `cooking egg` · `cooking on campfire` · `cooking sausages` · `counting money` · `country line dancing` · `covering something` · `cracking neck` · `crawling baby` · `crossing river` · `crying` · `curling hair` · `cutting nails` · `cutting pineapple` · `cutting watermelon` **D** — `dancing ballet` · `dancing charleston` · `dancing gangnam style` · `dancing macarena` · `deadlifting` · `decorating the christmas tree` · `digging` · `dining` · `disc golfing` · `diving cliff` · `dodgeball` · `doing aerobics` · `doing laundry` · `doing nails` · `drawing` · `dribbling basketball` · `drinking` · `drinking beer` · `drinking shots` · `driving car` · `driving tractor` · `drop kicking` · `drumming fingers` · `dunking basketball` · `dying hair` **E** — `eating burger` · `eating cake` · `eating carrots` · `eating chips` · `eating doughnuts` · `eating hotdog` · `eating ice cream` · `eating spaghetti` · `eating watermelon` · `egg hunting` · `exercising arm` · `exercising with an exercise ball` · `extinguishing fire` **F** — `faceplanting` · `feeding birds` · `feeding fish` · `feeding goats` · `felling tree` · `filling eyebrows` · `finger snapping` · `fixing hair` · `flipping pancake` · `fly tying` · `flying kite` · `folding clothes` · `folding napkins` · `folding paper` · `folding something` · `front raises` · `frying vegetables` **G** — `gargling` · `getting a haircut` · `getting a tattoo` · `giving or receiving award` · `golf chipping` · `golf driving` · `golf putting` · `grinding meat` · `grooming dog` · `grooming horse` · `gymnastics tumbling` **H** — `hammer throw` · `hand washing clothes` · `head stand` · `headbanging` · `headbutting` · `high jump` · `high kick` · `hitting baseball` · `hockey stop` · `holding something` · `holding something in front of something` · `holding something next to something` · `holding something over something` · `hoverboarding` · `hugging` · `hula hooping` · `hurdling` · `hurling sport` **I** — `ice climbing` · `ice fishing` · `ice skating` · `ironing` · `ironing clothes` **J** — `javelin throw` · `jetskiing` · `jogging` · `juggling balls` · `juggling fire` · `juggling soccer ball` · `jumping bicycle` · `jumping into pool` · `jumping jacks` · `jumping sofa` · `jumpstyle dancing` **K** — `kicking field goal` · `kicking soccer ball` · `kissing` · `kitesurfing` · `knitting` · `krumping` **L** — `laughing` · `laying bricks` · `laying stone` · `letting something roll down a surface` · `licking` · `lifting a surface with something on it` · `lifting hat` · `lifting something` · `lifting something up completely` · `lighting candle` · `lighting fire` · `lock picking` · `long jump` · `looking at phone` · `lunge` **M** — `making a cake` · `making a sandwich` · `making bed` · `making jewelry` · `making pizza` · `making snowman` · `making sushi` · `making tea` · `marching` · `massaging back` · `massaging feet` · `massaging legs` · `massaging persons head` · `milking cow` · `mopping floor` · `motorcycling` · `moving child` · `moving furniture` · `moving something across a surface` · `moving something away from something` · `moving something closer to something` · `moving something down` · `moving something up` · `mowing lawn` · `mushroom foraging` **N** — `news anchoring` **O** — `opening bottle` · `opening door` · `opening present` · `opening something` **P** — `packing` · `paragliding` · `parasailing` · `parkour` · `passing American football in game` · `passing American football not in game` · `peeling apple` · `peeling banana` · `peeling potatoes` · `petting animal not cat` · `petting cat` · `picking fruit` · `picking something up` · `pillow fight` · `pinching` · `plastering` · `playing accordion` · `playing badminton` · `playing bagpipes` · `playing basketball` · `playing bass guitar` · `playing cards` · `playing cello` · `playing chess` · `playing clarinet` · `playing controller` · `playing cricket` · `playing cymbals` · `playing didgeridoo` · `playing drums` · `playing flute` · `playing guitar` · `playing handball` · `playing harmonica` · `playing harp` · `playing ice hockey` · `playing keyboard` · `playing kickball` · `playing laser tag` · `playing monopoly` · `playing organ` · `playing paintball` · `playing piano` · `playing poker` · `playing recorder` · `playing saxophone` · `playing squash or racquetball` · `playing tennis` · `playing trombone` · `playing trumpet` · `playing ukulele` · `playing violin` · `playing volleyball` · `playing xylophone` · `plumbing` · `poaching eggs` · `poking something so it falls over` · `poking something so it slightly moves` · `pole vault` · `polishing furniture` · `polishing metal` · `popping balloons` · `pouring something into something` · `pouring something onto something` · `pretending to be a statue` · `pretending to pick something up` · `pretending to put something next to something` · `pretending to take something from somewhere` · `pretending to throw something` · `pull ups` · `pulling something from behind of something` · `pulling something from left to right` · `pulling something from right to left` · `pulling something onto something` · `pulling something out of something` · `pulling two ends of something so that it gets stretched` · `pumping fist` · `pumping gas` · `punching bag` · `punching person boxing` · `push up` · `pushing car` · `pushing something from left to right` · `pushing something from right to left` · `pushing something off of something` · `pushing something onto something` · `pushing something so it spins` · `pushing something so that it almost falls off but doesnt` · `pushing something so that it falls off the table` · `pushing something with something` · `pushing wheelchair` · `putting makeup on someone` · `putting on eyeliner` · `putting on foundation` · `putting on lipstick` · `putting on mascara` · `putting on shoes` · `putting something and something on the table` · `putting something behind something` · `putting something in front of something` · `putting something into something` · `putting something next to something` · `putting something on a flat surface without letting it roll` · `putting something on a surface` · `putting something on the edge of something so it is not supported and falls down` · `putting something onto a slanted surface but it doesnt glide down` · `putting something onto something` · `putting something similar to other things that are already on the table` · `putting something that cant roll onto a slanted surface so it slides down` · `putting something underneath something` · `putting something upright on the table` **R** — `reading book` · `reading newspaper` · `recording music` · `removing something spreading it apart from something` · `riding a bike` · `riding camel` · `riding elephant` · `riding mechanical bull` · `riding mountain bike` · `riding mule` · `riding or walking with horse` · `riding scooter` · `riding snow blower` · `riding unicycle` · `ripping paper` · `robot dancing` · `rock climbing` · `rock scissors paper` · `roller skating` · `rolling eyes` · `rolling pastry` · `rolling something on a flat surface` · `rope pushdown` · `rowing boat` · `running on treadmill` **S** — `sailing` · `salsa dancing` · `sanding floor` · `scrambling eggs` · `scuba diving` · `setting table` · `sewing` · `shaking hands` · `shaking head` · `shaking something` · `sharpening knives` · `sharpening pencil` · `shaving head` · `shaving legs` · `shearing sheep` · `shooting basketball` · `shooting goal soccer` · `shopping` · `shot put` · `shoveling snow` · `showing something behind something` · `showing something next to something` · `showing something on top of something` · `showing something to the camera` · `shredding paper` · `shuffling cards` · `side kick` · `sign language interpreting` · `singing` · `situp` · `skateboarding` · `ski ballet` · `ski jumping` · `skiing crosscountry` · `skiing mono` · `skiing slalom` · `skipping rope` · `skipping stone` · `skydiving` · `slacklining` · `slapping` · `sled dog racing` · `smoking` · `smoking hookah` · `snatch weight lifting` · `sneezing` · `sniffing` · `snorkeling` · `snowboarding` · `snowkiting` · `snowmobiling` · `somersaulting` · `spinning poi` · `spinning something so it continues spinning` · `spinning something that quickly stops spinning` · `spraying` · `springboard diving` · `squat` · `squeezing something` · `stacking something` · `stamping something` · `standing on hands` · `sticking something on something` · `sticking tongue out` · `stirring` · `stomping grapes` · `stretching arm` · `stretching leg` · `strumming guitar` · `stuffing something into something` · `surfing crowd` · `surfing water` · `sweeping floor` · `swimming backstroke` · `swimming breast stroke` · `swimming butterfly stroke` · `swing dancing` · `swinging legs` · `swinging on something` · `sword fighting` · `sword swallowing` **T** — `tackling football` · `tai chi` · `taking a shower` · `taking photo` · `taking something from somewhere` · `taking something out of something` · `tango dancing` · `tap dancing` · `tapping guitar` · `tapping pen` · `tasting beer` · `tasting food` · `tasting wine` · `texting` · `throwing axe` · `throwing ball` · `throwing discus` · `throwing knife` · `throwing something` · `throwing something against something` · `throwing something in the air and catching it` · `throwing something in the air and letting it fall` · `throwing something onto a surface` · `tickling` · `tipping something over` · `tipping something with something in it over so everything falls out` · `tobogganing` · `tossing coin` · `tossing salad` · `training dog` · `trapezing` · `trimming or shaving beard` · `trimming trees` · `triple jump` · `tumbling something downwards` · `turning something upside down` · `turning the camera left while filming something` · `turning the camera right while filming something` · `tying bow tie` · `tying knot not on a+rope` · `tying shoe laces` **U** — `uncovering something` · `unfolding something` · `using a power drill` · `using a wrench` · `using computer` · `using remote controller not gaming` · `using segway` **V** — `vault` · `visiting museum` **W** — `walking the dog` · `washing dishes` · `washing feet` · `washing hair` · `washing hands` · `water skiing` · `water sliding` · `watering plants` · `waving hand` · `waxing back` · `waxing chest` · `waxing eyebrows` · `waxing legs` · `weaving basket` · `welding` · `whistling` · `windsurfing` · `wiping something off of something` · `wrapping present` · `wrestling` · `writing` **Y** — `yawning` · `yoga` **Z** — `zumba` --- *This list reflects the deployed vocabulary and updates when the model checkpoint updates. Last verified: 2026-08-01.* --- # Error registry Every error code the API can return, with retry semantics. This registry is closed and append-only — codes are never removed or repurposed within v1. Machine-readable at `GET https://api.primateintelligence.ai/v1/errors`. **retryable** = safe to retry the same request unchanged (with backoff). **idem** = idempotent to retry with the same `Idempotency-Key`. ## HTTP errors ### `invalid_api_key` **HTTP 401** · retryable: **no** · idem: **no** API key is invalid. Check the key, env var, and whitespace. ### `key_revoked` **HTTP 401** · retryable: **no** · idem: **no** This API key has been revoked. Rotate or create a new key. ### `key_expired` **HTTP 401** · retryable: **no** · idem: **no** This API key has expired. Create a new key. ### `token_expired` **HTTP 401** · retryable: **no** · idem: **no** Client token has expired. Mint a new token from your server (client tokens are never refreshable). ### `insufficient_scope` **HTTP 403** · retryable: **no** · idem: **no** The key does not carry the scope required for this operation. ### `account_restricted` **HTTP 403** · retryable: **no** · idem: **no** Account is restricted. Complete signup/verification. ### `test_mode_only` **HTTP 403** · retryable: **no** · idem: **no** Operation not available in this key mode. ### `test_key_fixture_only` **HTTP 403** · retryable: **no** · idem: **no** Test keys can only analyze the provided fixture video. Create a live key to analyze your own uploads. ### `validation_failed` **HTTP 400** · retryable: **no** · idem: **no** Request validation failed. All violations are listed in error.errors[]. ### `unsupported_media_type` **HTTP 415** · retryable: **no** · idem: **no** Send video/mp4 or video/quicktime with a correct Content-Type. ### `payload_too_large` **HTTP 413** · retryable: **no** · idem: **no** Payload exceeds the 100MB multipart cap. Use the presigned upload path. ### `resource_not_found` **HTTP 404** · retryable: **no** · idem: **no** No such resource. IDs are account-scoped; check the ID and prefix. ### `resource_conflict` **HTTP 409** · retryable: **no** · idem: **no** Invalid state transition. GET the resource for its current state. ### `idempotency_key_reused` **HTTP 409** · retryable: **no** · idem: **no** Idempotency-Key was reused with a different request body. ### `idempotency_unavailable` **HTTP 409** · retryable: **yes** · idem: **yes** Idempotency store unavailable. Retry with the same key after Retry-After. ### `stream_already_active` **HTTP 409** · retryable: **no** · idem: **no** An active stream already exists on this account. End it (POST /v1/streams/{id}/end) or wait for it to finish. ### `upload_incomplete` **HTTP 400** · retryable: **no** · idem: **no** Upload incomplete or size mismatch. Re-create the video and complete within 24h. ### `video_unreadable` **HTTP 400** · retryable: **no** · idem: **no** Container/codec unsupported. Re-encode to H.264 MP4. ### `video_too_large` **HTTP 400** · retryable: **no** · idem: **no** Video exceeds the 2 GiB limit. ### `video_too_long` **HTTP 400** · retryable: **no** · idem: **no** Video exceeds the plan duration limit. Upgrade or trim. ### `url_fetch_failed` **HTTP 400** · retryable: **yes** · idem: **no** Source URL unreachable or timed out. ### `url_forbidden` **HTTP 400** · retryable: **no** · idem: **no** URL scheme or host blocked by the ingest policy. ### `prompt_empty` **HTTP 400** · retryable: **no** · idem: **no** Provide prompt or query. ### `prompt_too_long` **HTTP 400** · retryable: **no** · idem: **no** Prompt exceeds 2000 characters. ### `query_invalid` **HTTP 400** · retryable: **no** · idem: **no** Body fails the CompiledQuery schema; param names the field. ### `parse_failed` **HTTP 422** · retryable: **yes** · idem: **no** The compiler could not produce a query. Rephrase or send a structured query. ### `model_not_found` **HTTP 400** · retryable: **no** · idem: **no** Unknown model. See GET /v1/models. ### `model_deprecated` **HTTP 400** · retryable: **no** · idem: **no** Model is deprecated. Pin a supported version; see the changelog. ### `analysis_not_cancelable` **HTTP 409** · retryable: **no** · idem: **no** Analysis is already in a terminal state. ### `rate_limit_exceeded` **HTTP 429** · retryable: **yes** · idem: **yes** Rate limit exceeded. Back off per Retry-After. ### `concurrency_limit_exceeded` **HTTP 429** · retryable: **yes** · idem: **yes** Concurrent analysis limit reached. Wait for an active analysis to finish. ### `quota_exceeded` **HTTP 429** · retryable: **no** · idem: **no** Quota exceeded for a metered limit. See details.meter. ### `insufficient_credits` **HTTP 402** · retryable: **no** · idem: **no** Insufficient credits. Add credits or enable auto-refill. ### `grant_exhausted` **HTTP 402** · retryable: **no** · idem: **no** Free grant exhausted. Claim a full account via POST /v1/keys/request to continue. ### `capacity_exhausted` **HTTP 503** · retryable: **yes** · idem: **yes** Platform queue is full. Honor Retry-After. ### `sandbox_limit_exceeded` **HTTP 429** · retryable: **yes** · idem: **yes** Sandbox provisioning limit reached for this IP. Retry after the window resets, or sign up for a live key. ### `upgrade_limit_exceeded` **HTTP 429** · retryable: **no** · idem: **no** Too many upgrade attempts from this IP. Retry after the window resets. ### `device_code_expired` **HTTP 410** · retryable: **no** · idem: **no** Device code has expired or been consumed. Start a new request. ### `inference_unavailable` **HTTP 503** · retryable: **yes** · idem: **yes** Transient inference backend outage. Retry with backoff. ### `upstream_error` **HTTP 502** · retryable: **no** · idem: **no** Inference upstream returned an error. Message is sanitized — no internal paths or stack traces. ### `db_unavailable` **HTTP 503** · retryable: **yes** · idem: **yes** Transient database outage. Retry with backoff. ### `internal_error` **HTTP 500** · retryable: **yes** · idem: **yes** Internal error. Includes request_id; contact support if persistent. ### `webhook_endpoint_limit` **HTTP 400** · retryable: **no** · idem: **no** At most 10 webhook endpoints per account. ## Resource-level errors These never appear as HTTP responses — only in `analysis.error.code` / `video.error.code` on failed resources. ### `inference_error` Model failed on this input. Safe to create a new analysis; persistent failures on one video → support with request_id. ### `stuck_timeout` The platform lost the job past the sweep window. Not billed; resubmit. --- # Primate Intelligence API — Changelog Sorted newest-first. Agent-readable summary of shipped changes, broken by date. Full field contracts: [openapi.json](./openapi.json) · [agents.md](./agents.md) --- ## 2026-08-01 — feature — `unsupported_action` signal + supported-actions capability page (PRI-516) - **New optional result field `unsupported_action`**: when a prompt asks for something the model cannot detect/score (today: `indeterminate_reason: "unsupported_query_form"` — open-ended prompts), the result carries `unsupported_action: {reason, docs_url}`. `docs_url` points to the new [Supported actions](https://primateintelligence.ai/docs/supported-actions) page — the honest capability list including the full 532-class action vocabulary and known gaps (gestures, most vehicle actions). - **Additive only** — the field is absent (never null) on every other outcome; no existing request/response shape changed. - Applies at read time, so previously-created unsupported-form analyses also carry the signal on GET. --- ## 2026-08-01 — docs — preview status surfaced across the API surface (PRI-534) - **`/llms.txt`** gains a Model section naming `Darwin-preview-1.3B` (status preview, default), the deprecated `darwin-1.3` alias, what preview means, and a link to the new [Supported actions](https://primateintelligence.ai/docs/supported-actions) page. - **openapi.json**: the `Model.status` field and `GET /v1/models` description now explain the preview caveat; `model` request-field descriptions name the current default and its status. - **MCP `list_models` tool description** names the current default model and the preview caveat. - **agents.md** “What the Model Supports” opens with the model identity + preview meaning + supported-actions link. - No behavioral change — documentation/description strings only, all derived from the authoritative model registry (PRI-536). --- ## 2026-08-01 — rename — canonical model id is now `Darwin-preview-1.3B` (PRI-536) - **The public model id `darwin-1.3` is renamed to `Darwin-preview-1.3B`.** `GET /v1/models` now lists `Darwin-preview-1.3B` (status `preview`, default) with an `aliases: ["darwin-1.3"]` field. - **No breaking change**: requests sending `model: "darwin-1.3"` (any casing) or `"latest"` are still accepted — the server resolves deprecated aliases to the canonical id. Every response (`analysis.model`, `stream.model`) now emits `Darwin-preview-1.3B`, including for analyses/streams created before the rename. - `GET /v1/analyses?model=` accepts either spelling and matches the same rows. - Internally the identity now lives in a single authoritative model registry — future model upgrades bump one file and the catalog/defaults/docs follow. --- ## 2026-07-31 — docs/SDK/MCP — DX artifact chain for the PRI-496 public capabilities (PRI-530) - **Docs**: the mid-stream `update_prompt` → `prompt_updated` protocol is now fully documented (agents.md streaming lifecycle step 5 + openapi POST /v1/streams description), alongside the `status` event vocabulary (`prompt_context | combined_prompt | recalculating`) and the narrative opt-ins. `/llms.txt` gains a Live-narrative section and the client-token (pvct_) line. - **TypeScript SDK `@primate-intelligence/sdk@0.3.0`**: real `Narrative` type (was a `{text?}` stub), `StreamCreateParams.options.narrative` + `recording`, `Stream.recording`, `streams.recording()`, `videos.listAnalyses()`, `analyses.rerun()`, `Analysis.rerun_eligible`, `Video.media`, `StreamDetection.narrative_update`, `connectStream` `onNarrative`/`onStatus` callbacks + `StreamStatusMessage`. - **Python SDK `primate-intelligence==0.3.0`**: `streams.recording()`, `videos.list_analyses()`, `analyses.rerun()`; docstrings for the narrative opt-ins and the update_prompt protocol. - **MCP server 0.3.0** (hosted `/mcp` + `@primate-intelligence/mcp`): 13 tools — adds `get_video` (status + media playback URL), `list_video_analyses`, `rerun_analysis`; `create_analysis` gains `narrative: true`. Output schemas corrected (`narrative {status, entries}`, `Video.media`). - **Samples**: S6 (TS) and S7 (Python/aiortc) now exercise narrative, status events, mid-stream `update_prompt`, and (S6) session-recording retrieval. - All shapes verified against the live dev API (recorded narrative stream with status events + update_prompt, 2026-07-31). No API behavior change — documentation/tooling parity only. --- ## 2026-07-31 — internal — web-app library routes accept public ids (PRI-496 Lane B) - Internal-only (first-party web app; NOT part of the published contract): `GET /v1/user-videos` list rows now carry `public_id` (`video_…`) + `status`; the `/v1/user-videos/:id…` routes accept either the internal UUID or the public `video_` id; `PUT …/results` resolves public `an_` analysis ids to the internal job FK. Enables the web app's cutover to public `POST /v1/analyses` re-runs. Public OpenAPI document unchanged. --- ## 2026-07-31 — fix — narrative-opted-in public streams now activate the engine's narrative path (PRI-510/PRI-496) - Public streams created with `options: {narrative: true}` now send the engine-config serialization (`narrative_enabled: true`) to the inference engine — previously the engine's narrative path never activated for public streams, so `narrative_update` objects (and the new `status` events) could never appear even with the feature enabled. Non-opted-in streams are unchanged. Public prompt echo now strips ALL internal serializations (byte-identical up to the first `|||`). --- ## 2026-07-31 — fix — stream recording key stored verbatim from the sidecar (PRI-496) - The public recording linkage now stores the sidecar-reported S3 key **verbatim** (`{env}/recordings/…`) instead of stripping the env prefix. The sidecar's env label is the authoritative namespace of the object it wrote; the API's `NODE_ENV`-derived prefix disagrees on dev (Railway builds run `NODE_ENV=production`), which made dev recording URLs point at `prod/`. Caught in dev verification; prod behavior unchanged (labels agree there). --- ## 2026-07-31 — additive — free re-runs for platform-incident failures (PRI-493) + demo videos documented (PRI-496) - **Failed analyses now carry `rerun_eligible: boolean`** — `true` when the failure occurred during a declared platform incident (or ops flagged it). Field is present only on `failed` analyses; absent elsewhere. Additive. - **New `POST /v1/analyses/{id}/rerun`** — creates a **fresh analysis at no charge** from an eligible failed one (same video, prompt/query, model, options; new id; `usage` stays null — never billed). One free re-run per failed analysis; ineligible → `409 rerun_not_eligible` (new registry code). If the re-run's dispatch fails (503) the free re-run is NOT consumed. Normal capacity/brownout gates still apply. - **`GET /v1/demo-videos` is now documented** in openapi.json/agents.md as a public (unauthenticated) onboarding surface: curated sample videos with precomputed results per example prompt. Documented subset is the stable contract. - Decisions recorded for the remaining small gaps: stream limits already ship in the `POST /v1/streams` response (`limits`); `/v1/feedback` and `/v1/prompts` remain console-internal. --- ## 2026-07-31 — additive — stream session recordings + public status events (PRI-496) - **`POST /v1/streams` accepts `recording: true`** (optional boolean, default false): opt into retrieval of the server-side session recording. The stream then carries a new nullable `recording: {status}` object (`recording` · `available` · `failed` · `none`); streams that don't opt in keep `recording: null` — no existing field changed. - **New `GET /v1/streams/{id}/recording`** — returns `{object: "stream_recording", stream_id, url, expires_at, content_type, container}` with a **signed playback URL, fresh on every GET, 1-hour TTL** (sign-on-read, same semantics as `Analysis.artifacts`). The recording is a raw H.264 Annex-B elementary stream — lossless remux to MP4 with `ffmpeg -i in.h264 -c copy out.mp4`. 404 when recording wasn't enabled; 409 while the stream is active, when capture failed, or when nothing was stored. - **New additive `status` server→client event on the `/v1/streams/{id}/signal` WS**: `{type: "status", status: "prompt_context"|"combined_prompt"|"recalculating", message?}` — the narrative-ordering vocabulary, forwarded ONLY for streams that opted into `options: {narrative: true}` (and only while the narrative feature flag is enabled). Closed set at the edge; internal serializations are stripped from `message`. Existing vocabulary (`ready/queued/answer/ice/live/result/prompt_updated/metering/warning/end/error/pong`) is untouched; non-opted-in streams see zero change. --- ## 2026-07-31 — additive — video library parity: source playback URL + per-video analysis history (PRI-496) - Fixture videos (the sandbox plane's canned sample) return `media: null` — they are pseudo-objects with no real S3 source; a signed URL would 404. - **`Video.media` is now populated** on `ready` videos: `{url, expires_at}` — a signed playback URL for the original source video, generated **fresh on every read** with a 1-hour TTL (sign-on-read, same semantics as `Analysis.artifacts`). Re-fetch the video for a new URL after expiry; old videos always stay playable. `media` is `null` for non-ready videos and rows without a stored source object. New nullable field — additive, no existing field changed. - **New `GET /v1/videos/{id}/analyses`** — lists every analysis run against a video (newest first), same list envelope + filters (`status`, `limit`, `starting_after`) as `GET /v1/analyses`. Resource-nested spelling of `GET /v1/analyses?video_id={id}`; both work. This plus `media` plus `artifacts` gives the public plane full video-library semantics (list, replay, per-video result history). --- ## 2026-07-30 — breaking (gated) — GitHub OAuth gate on the free grant upgrade (PRI-479) - **`POST /v1/keys/upgrade` now requires a verified GitHub account** when the gate is enabled (it is, on dev + prod). Calling without a `github_token` returns **`403 github_verification_required`** whose `details` carry the full device-flow bootstrap: `client_id`, `device_code_url`, `access_token_url`. Complete GitHub's standard device flow (no scopes requested — identity only) and retry with `{"github_token": "gho_..."}`. - **One free grant per GitHub account, ever.** The numeric GitHub account id is stamped on the user row (DB unique index); a GitHub account that already claimed its grant gets **`409 github_account_already_used`**. The OAuth token itself is verified with GitHub and discarded — never stored. - New error codes in the §15 registry: `github_verification_required` (403), `github_token_invalid` (401), `github_account_already_used` (409). - Why: the upgrade was IP-rate-limited only — a key-churning agent rotating IPs could harvest unlimited 6,000s GPU grants. GitHub's own abuse controls now bound free grants at one per real account; IP caps remain as secondary defense. - Sandbox provisioning (`POST /v1/sandbox`) and the billed device-code claim flow (`POST /v1/keys/request`) are unchanged. --- ## 2026-07-30 — fix — internal/fixture accounts excluded from product metrics (PRI-472) - **New `users.is_internal` flag** (migration 036: `boolean NOT NULL DEFAULT false` + partial index + pre-filtered `analytics_users` view). Backfilled for the Anthropic directory-review fixture, all loadtest accounts, and E2E fixtures. - **All admin analytics endpoints now exclude internal accounts**: `/v1/admin/analytics/growth` (totals, signups, activation, W1 retention, MRR/ARR/paying users, North-Star minutes), `/v1/admin/pmf` (signups_total, activation funnel, repeat-user retention), `/v1/admin/analytics/credits` (card attach rate, refill conversion, API activation), `/v1/admin/analytics/verticals` (user vertical/replacing breakdowns). - **Clerk `user.created` webhook** now propagates `public_metadata.internal_fixture: true` → `users.is_internal`, so future fixture accounts are excluded at creation. - Historical note: metric values step down slightly vs. previous reports (fixture activity was previously counted — e.g. 105 loadtest users on prod). This is a correction, not a regression. - Admin response fields only — no public API schema change. --- ## 2026-07-30 — additive — API host agent onramp: GET / index + GET /robots.txt (PRI-492) - **`GET /` now returns a JSON index** instead of a 404 error envelope: `{name, description, openapi, docs, llms, agents, changelog, mcp, sandbox, health}` — an agent probing the API host root gets pointers to every machine-readable surface (OpenAPI spec, llms.txt, agents.md, MCP endpoint, one-POST sandbox key). - **`GET /robots.txt` now returns 200 text/plain** (allow all) instead of the 404 JSON envelope, and points crawlers at `/llms.txt`, `/v1/openapi.json`, and `/docs/agents.md`. - Both endpoints are unauthenticated. No existing route changed; no schema change. --- ## 2026-07-30 — fix — GET /v1/models: darwin-1.3 capabilities.streaming corrected to true (PRI-496) - The model catalog still said `capabilities: {streaming: false}` from before real-time streams shipped. Streaming has been live for weeks (`POST /v1/streams`); the flag now says so. Data correction only — no schema change. --- ## 2026-07-30 — additive — Analysis.artifacts now populated: annotated result video URL (PRI-496) - **`GET /v1/analyses/{id}` (and list) now populates `artifacts`** for completed analyses that produced an annotated result video: `{annotated_video_url, expires_at}` per the existing `ArtifactsSchema`. The URL is a **fresh 1-hour signed CDN link generated on every read** — when it expires, re-fetch the analysis for a new one (old results always stay retrievable). Analyses without an annotated video (and all non-completed statuses) continue to return `artifacts: null`. - The schema stub has been in the contract since P3 — this makes it live. Zero breaking changes; agents that treated `artifacts` as always-null should start reading it. --- ## 2026-07-30 — additive — narrative over the public API: options.narrative on analyses + streams (PRI-510) - **`POST /v1/analyses` (and `/batch`) `options.narrative: true` is now honored.** The completed analysis carries a populated `narrative` object per the existing `NarrativeSchema`: `{status: "generating"|"ready"|"failed", entries: [{t_s, text}]}` — timestamped event sentences generated after completion (poll `GET /v1/analyses/{id}`; the `analysis.completed` webhook may still show `status: "generating"`). Omitted/false preserves the previous behavior exactly (`options: {narrative: false}`, `narrative: null`). Narrative is **included in the analysis price** — no surcharge, no usage-shape change. - **`POST /v1/streams` accepts `options: {narrative: boolean}`** (previously rejected as an unknown field). Opted-in streams receive a `narrative_update` object (`{t_s, text}`) on result-frame detections whenever the engine produces a new sentence — event-driven, not per frame; the key is absent on frames without one. Disabled/omitted streams never see it. Terminal `results_summary.last_detections` continues to strip it (frozen shape). - **Test mode (`pv_test_`)**: opted-in canned analyses return a deterministic canned narrative (`status: "ready"`) — fixture plane, zero quota. - The schema fields (`options.narrative`, nullable `narrative`, `NarrativeSchema`) were already in the contract as forward-compatible stubs — this release makes them live. Zero breaking changes (oasdiff clean). --- ## 2026-07-30 — internal — inference-WS replacement-storm alerts now reach Sentry (PRI-418); load-test metric renamed to metadata_ratio (PRI-360) - **Replacement-storm detector alerts to Sentry** (PRI-418): when inference-WS replacements exceed the threshold (default 10/60s), the existing error-level structured log is now also sent via `Sentry.captureMessage` at error level, so storms like the 2026-07-02 duplicate-sidecar incident page instead of only landing in stdout. Cooldown unchanged (max one alert per 5 min). No public API contract change. - **Load-test harness: `metadata_ratio` metric added** (PRI-360): the k6 shared metrics library gains a `metadata_ratio` Rate (fraction of incoming sidecar WS messages that are `metadata` frames), wired into `09-stream-baseline.js`. Replaces the informal "ack ratio" name — the counted frames are metadata, not acks. Tooling-only; no runtime behaviour change. --- ## 2026-07-29 — additive — batch discount is config-driven and served by GET /v1/credit-pricing (PRI-508) - **`GET /v1/credit-pricing` now returns `batch_discount_pct`, `batch_min_prompts`, and `batch_max_prompts`.** The `POST /v1/analyses/batch` discount (each prompt after the first) was a hardcoded constant that could drift from the documented "50%"; it now lives in the same `credit_price_config` row as every other price knob and is served publicly here. **Read the discount from this endpoint instead of hardcoding it** — a pricing change is a config update, not an API version bump. Current values match shipped behaviour exactly: `batch_discount_pct: 50`, `batch_min_prompts: 2`, `batch_max_prompts: 10`. - **`discount_pct` in batch responses** (`analysis_batch.pricing`, `analysis_batch_preview.prompts[].discount_pct`) now reflects the configured value (still `50` today). The preview per-prompt field is documented as an integer rather than the literal set `0 | 50` — `0` still means the full-price first prompt. - No behaviour change at current config: batch pricing, bounds, and copy are byte-identical until the config row changes. --- ## 2026-07-29 — fix — /v1/test-fixture video URL follows the app.* replatform; anonymous sandbox IP cap now per-client - **`GET /v1/test-fixture` `test_video_url` updated** to `https://app.primateintelligence.ai/empty-state/forklift-demo.mp4`. The web replatform moved the product SPA (which serves the fixture file) to `app.primateintelligence.ai`; the old apex path stopped resolving when the apex became the marketing site. If you pinned the old URL in CI, update it — the file bytes are unchanged. The canonical pinned demo asset `https://primateintelligence.ai/demos/empty-state/forklift-demo.mp4` is unaffected. - **`POST /v1/sandbox` IP cap fixed to key on the real client IP.** Previously the daily provision cap could key on a shared proxy hop, which made the limit far stricter than documented for users behind the same edge. No contract change — same `sandbox_limit_exceeded` error, same documented 3/day per-IP semantics, now enforced per actual client. --- ## 2026-07-29 — docs-only — POST /v1/parse documented in openapi.json; GET /v1/analyses status-filter vocabulary documented - **`POST /v1/parse` is now in the OpenAPI document** (PRI-496 P0). The endpoint has been live for months (stateless prompt compiler: `{prompt}` → `{compiled_query, parse_mode: "haiku"|"heuristic"}`; auth optional; 30 req/min per IP; errors `empty_prompt` 400 / `parse_failed` 422 / `parse_unavailable` 503) but was undocumented — for agents, undocumented is nonexistent. No behavior change. Also added to the `/llms.txt` key-endpoints list. - **`GET /v1/analyses` `status` filter vocabulary documented**: `queued | preparing | analyzing | rendering | completed | failed | canceled` (§13.2). The filter has always worked (unknown values → 400 `validation_failed`); its accepted values were just not written down. Documented in the parameter description — the machine schema stays `type: string`, so nothing tightens contractually. Use `status=analyzing` for currently-running analyses. - Zero contract changes — the public surface is frozen; this is documentation of already-shipped behavior. --- ## 2026-07-28 — additive — SDK 0.2.0 (TS + Python) and MCP 0.2.0: full parity with the July API rounds - **TypeScript SDK `@primate-intelligence/sdk@0.2.0`** (breaking — type corrections): `AnalysisUsage` fixed to the real contract `{billed_seconds, credit_balance_after}`; `StreamEndReason` fixed (adds `canceled`/`ice_failed`/`media_timeout`, drops never-returned `client_disconnect`); `video_duration_s` nullable; `Analysis` gains `livemode` + `origin`. New: `client.credits.retrieve()` (`GET /v1/credits`), `client.creditPricing.retrieve()`, `analyses.validateOnly()`, `analyses.createBatch()`/`validateBatch()`, `detected_count` + `indeterminate_reason` on results, typed streaming contract in `/browser` (`session_remaining_s` metering, `no_media_frames` warning union, typed detections, `failure_diagnostic`, typed `results_summary`). Migrate off the `balance_s`/`results_summary.frames` aliases before their ~2026-08-28 removal. - **Python SDK `primate-intelligence==0.2.0`** (additive): `client.credits` + `client.credit_pricing` resources, `analyses.validate_only()`, `analyses.create_batch()`/`validate_batch()`, docstrings updated to the real result/streaming contracts. - **MCP 0.2.0 (hosted `/mcp` + npm `@primate-intelligence/mcp`)**: three new tools — `validate_analysis` (free dry-run with cost estimate), `create_analysis_batch` (batch discount), `get_credits` (transaction ledger). `get_analysis`/`wait_for_analysis` descriptions now document `detected_count`, `indeterminate_reason`, nullable duration, and the usage snapshot. The npm package catches up with the hosted server: every tool declares `outputSchema` and returns `structuredContent`; `wait_for_analysis` returns the `{analysis, retry}` envelope (the legacy `_mcp_note` field merged into the analysis resource is gone from the npm package too). --- ## 2026-07-28 — failure_diagnostic hint wording: customer-appropriate, no internal references - **`failure_diagnostic.hint` (on `end_reason: "ice_failed"`) reworded.** The hint previously referenced an internal server environment variable — meaningless (and confusing) to API consumers. It now states plainly which side the fault is likely on and what to do: when the server did not advertise a publicly reachable network candidate → *“server-side configuration issue — nothing to change on your end; retry, and contact support if it persists”*; when the server did advertise public candidates → *“check your network allows UDP or TCP/443 outbound and that your client uses the TURN servers in `ice_servers`”*. Field shape unchanged (`{local_candidates, hint}`); only the hint text changed. Same sweep reworded the transient status messages sent on the signaling WS (`inference_unavailable`, capacity/restart rejections) to remove internal jargon. --- ## 2026-07-28 — Metering tick + results_summary field renames (deprecation aliases until ~2026-08-28) - **Metering tick: `balance_s` renamed to `session_remaining_s`** (Streaming DX Finding 2). The value was always the seconds remaining in **this session's** credit reservation (the session cap) — the old name read like account credit balance, which lives at `GET /v1/billing/credits` (`balance_seconds`). Every tick now carries BOTH `session_remaining_s` (canonical) and `balance_s` (deprecated alias, identical value). **`balance_s` will be removed ~2026-08-28** — migrate reads to `session_remaining_s`. - **Terminal `results_summary`: `frames` renamed to `result_frames`** (Streaming DX Finding 3). Results are **sampled**, not per-source-frame (adaptive inference cadence, roughly every 8th source frame under load), so the count is the number of result **events emitted** — `result_frames` says that; `frames` implied a source-frame count. Both fields present (identical value); **`frames` will be removed ~2026-08-28** — migrate reads to `result_frames`. - **Docs: result-sampling semantics now documented** in agents.md — `frame_num` on live results is the *source-frame index* (gaps between consecutive values are normal); `results_summary.result_frames` counts result events; expect roughly 8–15 results/s depending on load. - No behavior changes — both renames are additive aliases for 30 days, then the deprecated names drop. --- ## 2026-07-28 — No-media warning: the server now tells you when your media is the problem - **New signaling-WS event: `warning {code: "no_media_frames"}`** — when the WebRTC transport connects but no decodable video frame arrives within 5 seconds, the server pushes `{type:"warning", code:"no_media_frames", transport_connected:true, elapsed_s, packets_received, frames_decoded, hint}` (re-warned once at 15s, then quiet; canceled by the first decoded frame). Previously the server stayed silent for the full media-timeout window (~50s) with zero client-visible signal. `packets_received` distinguishes the two failure classes: `0` = your client is not sending media (track dead/muted); `>0` with `frames_decoded: 0` = media arrives but is not decodable (wrong codec, or a source producing no real frames — e.g. canvas capture without a draw loop). - **`end_reason: "media_timeout"` now carries the media counters in `failure_diagnostic`** when a connected-but-no-media session ends without ever going live. Billed 0, as before. - Existing `warning {remaining_s}` credit-exhaustion event is unchanged (distinguish by the `code` field — credit warnings have no `code`). --- ## 2026-07-28 — TURN-over-TLS (`turns:`) relay advertised for TLS-only egress networks - **ICE server list now always includes a `turns::443?transport=tcp` relay** in session credentials. Twilio's NTS token response carries `turn:` relays (UDP/TCP 3478, TCP 443) but no `turns:` URL, so clients behind TLS-only firewalls (datacenter/corporate egress allowing only TLS on 443) could never reach a relay and sessions failed with `ice_failed`. The API now synthesizes the `turns:` entry from the existing relay host, reusing its ephemeral credentials. No-op when the provider already returns a `turns:` URL or in STUN-only fallback. No request/response field changes — the `ice_servers` array simply gains one entry. --- ## 2026-07-28 — MCP tools declare outputSchema + structuredContent; wait_for_analysis timeout shape cleanup - **All 7 hosted MCP tools now declare an `outputSchema`** (in `tools/list` on `POST /mcp` and on the static server card at `/.well-known/mcp/server-card.json`). Schemas are generated from the same Zod DTOs that validate the public /v1 responses (`src/dto/schemas.ts`), so they cannot drift from the real contract. Per the MCP spec, every tool result now also carries `structuredContent` conforming to its declared schema (the SDK validates each result at runtime); the human-readable JSON text content is unchanged. - **`wait_for_analysis` response shape changed** (privacy-audit finding F-1): the tool now returns `{ analysis, retry }` — the analysis resource unmodified under `analysis`, and `retry: null` on terminal states or `retry: { reason: "timeout", note }` when the wait expired. The old behavior merged an undocumented `_mcp_note` field into the analysis object on timeout, mutating a documented resource shape. Agents reading the analysis should use `result.analysis` (or `structuredContent.analysis`). - **`query` field description** on analysis outputs now states explicitly that it is the compiled, deterministic interpretation of the caller's own prompt — a transparency feature (privacy-audit finding F-2, documentation only; no data change). - REST API surface unchanged — this release only affects the MCP endpoint metadata and the MCP-layer `wait_for_analysis` envelope. --- ## 2026-07-28 — OpenAI apps domain verification + explicit destructiveHint on read-only MCP tools - **New public endpoint: `GET /.well-known/openai-apps-challenge`** — returns the OpenAI apps domain-verification challenge token (`OPENAI_APPS_CHALLENGE_TOKEN` env var) as raw `text/plain`; 404 when unset. Unauthenticated by design — OpenAI's plugin/apps portal fetches it to verify domain ownership (PRI-475). Not part of the product API surface. - **MCP tool annotations: explicit `destructiveHint: false`** added to all five read-only tools (`get_analysis`, `wait_for_analysis`, `list_models`, `get_usage`, `get_test_fixture`) in both the live `tools/list` response and the static server card at `/.well-known/mcp/server-card.json`. Previously the hint was omitted (spec default is `true`), which OpenAI's tool scanner flagged. No changes to tool names, descriptions, or input schemas. --- ## 2026-07-28 — Streaming DX: server ICE fix + trickle, honest end reasons, unified result contract, public examples repo ### Transport (the critical fix) - **Server-side ICE candidates are now reachable from anywhere** — the streaming server advertises its public address as a host candidate and **trickles late-gathered srflx/relay candidates** to the client as `ice` messages after the answer SDP (previously candidates that outlasted the bounded gathering window were silently dropped, so relay-only clients — datacenters, CI, UDP-blocked networks, strict NATs — stalled in `ice: checking` until timeout while browser clients survived via peer-reflexive discovery). Keep consuming `ice {candidate}` messages after `answer`. - **Relay-only regression client in CI** — every deploy is tested with an aiortc client that offers only TURN relay candidates (TURN-over-TCP:443), the datacenter/edge-fleet profile. ### Honest `end_reason` (breaking-adjacent: two new enum values) - **New values: `ice_failed`, `media_timeout`** on `stream.end_reason` (WS `end {reason}` and the terminal Stream resource). A session that never went live is **never** `completed` — enforced server-side. - **New nullable field: `failure_diagnostic`** on the Stream resource — on `ice_failed`, carries `{local_candidates, hint}` (the server-side candidate summary) so transport failures are debuggable from the API alone. - Billing unchanged: never-live sessions bill exactly 0 (join/negotiation/queue free). ### Streaming result contract unified with file analyses (breaking for undocumented internals) - `result.detections[].answer` is now **lowercase `yes|no|indeterminate`** — the same enum as file-API results (was `"Yes"/"No"`, an internal casing leak; enums never vary by transport). - `result.detections[].prompt` now echoes **exactly what you submitted** — the internal `|||COMPILED_QUERY:` serialization no longer leaks into the public field. - Latency telemetry (`inference_ms`, `server_elapsed_ms`, per-hop p50s, per-stage `latency` breakdown) is grouped under a documented **`timing`** object. Internal scheduling fields (`queue_fill`, `queue_length`, `is_warm`, `video_enabled`) are no longer emitted. - Same normalization applies to `results_summary.last_detections` on the terminal resource. ### Docs & examples - **Public examples repo**: https://github.com/Primate-Intelligence/primate-examples — Python streaming client (aiortc, the "stream a file" recipe), browser webcam example, relay-only CI harness. The old private-repo examples link in the streaming guide is gone. - **agents.md gains a full Streaming section** — lifecycle, signaling protocol incl. server trickle, the streaming result contract, the honest end_reason vocabulary, the datacenter client profile, and billing semantics. --- ## 2026-07-27 — PRI-482 round 8: one-clock docs — canonical docs served by the API, website proxies them, hardened doc-sync CI gate, count-confidence semantics clarified ### Documentation architecture (the "separate clocks" fix) - **Single source of truth** — `docs/agents.md` and `docs/changelog.md` ship inside the API deploy artifact and are served at `https://api.primateintelligence.ai/docs/agents.md` / `/docs/changelog.md`. The website (`https://primateintelligence.ai/docs/agents.md` and `/docs/changelog.md`) now **proxies these API routes** instead of serving a hand-copied duplicate. Spec, quickstart, and changelog update atomically with the code they describe — there is no copy step left to go stale. - **RSS moves to the API** — `GET /docs/changelog.xml` is generated from `docs/changelog.md` at request time, so the feed can never drift from the changelog it is derived from. - **Hardened doc-sync CI gate** — a diff touching `src/routes/**`, `src/dto/paths.ts`, or billing/pricing services (`credit-ledger-service`, `billing-service`, `usage-meter-service`, `quota-service`) now fails CI unless **both** a docs surface (`docs/agents.md` or `docs/openapi.json`) **and** `docs/changelog.md` move in the same diff. Previously route changes required only one doc surface and billing changes required none. - **Nightly drift monitor** — an external probe diffs the website-served docs against the API canonical and alerts on any byte mismatch (belt-and-suspenders against pipeline-level failures CI can't see). ### Contract clarification - **Count-query confidence** — agents.md (Result Contract + Count queries sections) and `llms.txt` now state explicitly: for count queries, `confidence` applies to the detected count itself — it is the model's confidence that `detected_count` is the correct number, not merely that something was detected. --- ## 2026-07-27 — PRI-482 round 7: batch validate_only, enriched batch pricing, llms.txt batch listing, docs-atomicity CI gate, result-contract confidence cross-reference ### New capabilities - **`POST /v1/analyses/batch` with `validate_only: true`** — dry-run for batch: parses all prompts, returns `analysis_batch_preview` (HTTP 200) with per-prompt `query`, `parse_mode`, `assessable`, `estimated_seconds`, `estimated_cost_usd`, and `discount_pct`; plus batch-level totals `estimated_total_seconds` and `estimated_total_cost_usd`. No credits reserved, no jobs created. Estimates are `null` when the video has no known duration. - **Enriched batch pricing on real create** — `POST /v1/analyses/batch` response `pricing` block now includes `estimated_total_seconds` and `estimated_total_cost_usd` (null when duration unknown), so callers can see the expected cost without a separate `validate_only` round-trip. ### Docs & CI - **`llms.txt` batch endpoint** — `POST /v1/analyses/batch` added to the `## Key endpoints` section. - **Docs-atomicity CI gate** — `scripts/changelog-gate.ts` now enforces a second rule: any diff touching `src/routes/**` or `src/dto/paths.ts` must also update `docs/agents.md` or `docs/openapi.json`. Bypass with `[skip-docs-gate]` in a commit message. - **Result Contract confidence cross-reference** — `result.confidence` bullet in agents.md "Result Contract" section now notes that for count queries confidence reflects detection certainty (not count certainty), with a link to the Count queries section. - **agents.md batch section** — added `validate_only` documentation with worked JSON example. - **OpenAPI** — `validate_only` added to `CreateAnalysisBatchRequest`; `AnalysisBatchPreview` response schema added; `AnalysisBatch.pricing` updated with estimated fields. --- ## 2026-07-27 — PRI-482 round 6: batch discounts real, confidence semantics, key lifecycle in OpenAPI, 404 envelope, ledger source_type, historical balance freeze ### New endpoints - **`POST /v1/analyses/batch`** — submit 2–10 prompts against the same video in one request. First prompt billed at full price; each additional prompt billed at **50%**. Credits reserved per-prompt at the discounted rate (observable directly in `GET /v1/credits` ledger). Response includes `analyses[]` array + `pricing: {full_price_prompts, discounted_prompts, discount_pct: 50}`. Registered in OpenAPI spec and documented in agents.md "Batch analyses & discounts" section. - **`POST /v1/keys/upgrade`** and **`POST /v1/keys/request` / `GET /v1/keys/request/{code}`** now registered in `docs/openapi.json`. These routes existed but were absent from the spec — they are the first two calls any agent makes. ### Error handling - **404 catch-all** — unknown routes previously returned the framework-default `{"message":"Route ... not found"}`. They now return the standard typed error envelope (`code: "resource_not_found"`, same shape as all other errors). - **`POST /v1/analyses` with `prompts` array** now returns a helpful validation error pointing to `POST /v1/analyses/batch` instead of a generic "Unknown field". ### Billing fixes - **`source_type: "analysis"` for analysis debits** — credit ledger rows for public-API analyses were labelled `source_type: "upload"`. They are now labelled `source_type: "analysis"` (new DB constraint value). Migration 077 backfills existing reservation and transaction rows in dev. Agents reading `GET /v1/credits` will see `source_type: "analysis"` on analysis debits going forward. - **Historical `credit_balance_after` frozen** — analysis rows settled before migration 075 had a null `balance_after_seconds` snapshot, causing `credit_balance_after` to drift as later analyses ran. Migration 078 stamps those rows with the current account balance, freezing the value. `credit_balance_after: null` (`snapshot_unavailable`) now only appears when the account balance itself is unavailable. ### Docs & contract - **Confidence semantics for count queries** — added a paragraph to agents.md "Count queries — the contract" section: `confidence` is the aggregate detection confidence (max of per-term values, [0,1]); forced to 0 when `answer: "indeterminate"`; `≥0.7` reliable, `0.4–0.69` marginal, `<0.4` unreliable. - **Changelog link in agents.md** — added a dedicated "Changelog" section linking `/docs/changelog.md`. - **`snapshot_unavailable` semantics** — `usage.credit_balance_after: null` documented in agents.md Result Contract. - **Walking fixture section** added to agents.md "Action pipeline verification" (clip URL pending upload). ### Migrations applied to DEV - `077_analysis_source_type.sql` — adds `'analysis'` to `credit_reservations.source_type` constraint; backfills existing rows. - `078_freeze_historical_balance_after.sql` — stamps null `balance_after_seconds` on settled reservations with current account balance. --- ## 2026-07-27 — PRI-480 round 5: agent DX — versioned responses, immutable billing snapshot, truthful origin, pricing discovery ### Breaking changes - **`analysis.origin` enum changed: `user` → `api`, new value `console`.** The round-2 entry below introduced `origin: "user" | "system"` derived at read time from `source_type` — that heuristic misclassified console (dashboard) uploads as `system` and recorded nothing. `origin` is now stamped truthfully at job creation: `api` (public API), `console` (dashboard upload), `system` (internal — auto-probe, fixture seeding, run_prompt). Legacy rows without the column fall back to the heuristic (`public_api*` → `api`, `upload` → `console`, else `system`). The field shipped yesterday with near-zero consumers. ### New fields & headers - **`X-Api-Version` response header** — every response (including errors) carries `+`. If the value changes mid-session, a deploy happened — re-check this changelog before assuming a bug. - **`usage.credit_balance_after` is now immutable.** Previously it re-read the live account balance at GET time, so an old analysis's value mutated as later analyses ran. It is now a point-in-time snapshot stamped at settlement (`credit_reservations.balance_after_seconds`). Analyses settled before this deploy fall back to the live balance (previous behaviour). ### Billing invariant - **System-initiated analyses are never billed.** `origin: "system"` jobs (e.g. internal run_prompt) never reserve or settle credits — enforced by a CI test auditing every `reserveCredits` call site. ### Docs & discoverability - **`GET /v1/credit-pricing`** (public, no auth) is now in the OpenAPI spec, the [agents.md](./agents.md) Pricing section, and `/llms.txt`. Returns `price_per_second_cents`, `signup_grant_seconds`, `card_grant_seconds` — cost of an analysis = `usage.billed_seconds × price_per_second_cents`. - **Count-query contract** section added to [agents.md](./agents.md): `detected_count` is only meaningful when `answer` is determinate; success/failure examples are schema-validated in CI. - **Changelog is now CI-enforced** — any push/PR diff touching `src/**` must also update this file with a well-formed topmost `## YYYY-MM-DD — ` entry (`scripts/changelog-gate.ts` + `changelog-gate.yml`). - **Late note (shipped earlier today in commit df7f0ce without an entry):** `GET /docs/agents.md` and `GET /docs/changelog.md` are now served publicly — `/llms.txt` referenced both but they 404'd. - Changelog re-sorted newest-first (the 2026-07-24/22/18 entries were filed below 2026-07-09). --- ## 2026-07-27 — PRI-480 round 2: result invariants, synchronous metadata, billed_seconds, origin ### New fields - **`analysis.origin`** — `"user"` (submitted via the public API) or `"system"` (generated internally, e.g. auto-probe or fixture seeding). Agents should normally only act on `origin: "user"` analyses. ### Behaviour fixes - **Result consistency invariants.** A completed analysis returning `detected_count: 5` + `answer: "indeterminate"` + `indeterminate_reason: "nothing_detected"` is now repaired at the API boundary: (a) `nothing_detected` forces `detected_count` to 0, (b) `indeterminate` forces `confidence` to 0, (c) determinate `yes`/`no` answers never carry an `indeterminate_reason`. - **Video metadata populated synchronously on `/complete`.** `POST /v1/videos/:id/complete` now awaits the ffprobe enrichment (15 s timeout) before building the response, so `duration_s`, `width`, `height`, and `fps` are present in the response body and in immediate `GET /v1/videos/:id` reads. On timeout the endpoint returns the row as-is and enrichment continues in the background. - **`billed_seconds` on list analyses now correct.** The `GET /v1/analyses` list was querying `credit_reservations.job_id` (a column that does not exist). Fixed to join via `jobs.reservation_id → credit_reservations.id`, matching the single-GET path. `usage.billed_seconds` is now populated on all settled list items. ### New endpoints - **`GET /llms.txt`** — plain-text LLM index following the llms.txt convention. No auth required. Lists the API base URL, `/docs`, `/openapi.json`, and changelog. Lets AI coding assistants discover the API surface without a browser. ### Docs - **Query-type maturity note** added to [agents.md](./agents.md): presence and counting queries are mature/GA; action and temporal queries are beta (duration-sensitive); open-ended prompts are rejected with `unsupported_query_form` rather than billed. --- ## 2026-07-27 — PRI-480: API hardening, credits, and fixture parity ### Breaking changes None. ### New fields - **`result.video_duration_s`** — now `number | null` (was always `number`). Returns `null` when the source video duration could not be determined (short uploads, metadata-less files). Agents must null-check this field. - **`result.indeterminate_reason`** — why the answer is `indeterminate`. Enum: - `low_confidence` — model saw something but confidence was below threshold; try a more specific prompt. - `nothing_detected` — all confidence scores were 0; no subject detected in any frame. - `unsupported_query_form` — open-ended or freeform query; rephrase as a closed yes/no or count question. - `duration_mismatch` — inference result duration disagrees with source video duration (>20%); result is untrusted. - **`result.detected_count`** — integer count for count-intent queries ("how many X…"). `null` on yes/no queries. ### New endpoints - **`GET /v1/credits`** — returns `{ balance_seconds, grant_seconds, used_seconds, transactions: [...] }`. Per-analysis transaction history. Lets agents audit every charge independently. No parameters needed; key-scoped. ### New parameters - **`POST /v1/analyses` → `validate_only: boolean`** — parse + estimate cost, no GPU, no credits deducted. Returns `AnalysisPreview` with `estimated_cost_seconds`, `query_type`, and `answerability` signal. Use before committing to an analysis. ### Behaviour fixes - **Action pipeline duration now deterministic.** The action query path (`how many X…`) was returning GPU wall-clock time as `video_duration_s` instead of source video duration. For a 6s video this produced 40–75s billing. Root cause: `duration_seconds` was absent from the action result dict in the inference server; the API fell back to GPU wall-clock. Fixed in inference I-1 (inference commit `4e0825a`). - **`detected_count` now populates for action queries.** The count was computed by inference but not forwarded in the callback payload. Fixed in inference I-2. - **`indeterminate_reason` forwarded from inference.** When inference explicitly sets a reason, the API now respects it rather than re-classifying. - **Duration invariant cross-check.** If inference returns a duration >20% different from the admitted source duration, the API flags the result `indeterminate` with `indeterminate_reason: duration_mismatch` rather than silently returning wrong timing. - **Open-ended query gate.** Submitting a query the model cannot answer (open-ended form) now returns `answer: indeterminate`, `indeterminate_reason: unsupported_query_form` immediately, at zero cost, instead of burning GPU credits and returning an unreliable result. - **Video metadata probed at upload-complete.** `duration_seconds`, `width`, `height`, `fps` are populated on the video object immediately after `POST /v1/videos/:id/complete`. Fire-and-forget ffprobe enrichment runs after response is sent for non-faststart files. ### Test fixture update `GET /v1/test-fixture` now returns a `fixtures[]` array with two entries: - `presence` — forklift yes/no (existing fixture, unchanged) - `action` — forklift count query; assert `detected_count >= 1` Legacy flat fields (`test_video_url`, `test_prompt`, `expected_answer`, `expected_confidence_min`) preserved for backwards compatibility. --- ## 2026-07-24 — PRI-477: billing cap, detected_count, error envelope ### New fields - **`result.detected_count`** — integer count on count-intent queries (object path). - **`analyses.list` response** — includes `billed_seconds` per analysis. - **Error envelope** — all errors now return `{ error: { code, message, request_id } }` (was bare `{ error, message }`). ### Behaviour fixes - **Billing cap** — per-analysis billing is capped at the source video duration. A slow GPU run (warmup, recompile) cannot overbill beyond the video length. - **`answer: indeterminate`** — when confidence is 0 and all term confidences are 0 the answer is now `indeterminate` instead of `no`. A `no` that can't be substantiated is indistinguishable from a model that saw nothing. - **`usage` deprecated** — the `GET /v1/usage` response is kept but `usage` inline on analysis objects has moved to `GET /v1/credits`. --- ## 2026-07-22 — PRI-476: inline usage, url-ingest probe, liveProgress ### New fields - **`analysis.usage`** — inline `{ billed_seconds, credit_balance_after }` on completed analyses. Agents no longer need a second GET /v1/usage round-trip. - **URL-ingest video probe** — `POST /v1/videos` with `source_url` now runs the same metadata probe that upload-complete uses. --- ## 2026-07-18 — Public API v1 GA Initial public release of the REST API with: - `POST /v1/sandbox` — instant `pv_test_` key, no email, no card - `POST /v1/keys/upgrade` — upgrade to `pv_live_` + 6,000s free grant - `POST /v1/videos` — presigned upload or URL ingest - `POST /v1/analyses` — submit prompt + video → analysis - `GET /v1/analyses/:id` — poll or use `Prefer: wait=60` - `GET /v1/analyses` — paginated list - `GET /v1/credits` — balance + transactions - `GET /v1/test-fixture` — stable CI fixture --- ## 2026-07-12 — Credit notifications, ToS consent, CORS ### New features - **Expiring-credits reminder email** — sent once per credit lot, 7 days before expiry. Requires opt-in notification prefs. - **Low-balance email** — fires once per episode (was: every 24h). ### Behaviour fixes - **Admin API-key mint/rotate** now stamps implied Terms of Service acceptance. - **Admins exempt from ToS gate** on stream and upload endpoints. - **CORS preflight** now allows `X-Primate-Client` header (dashboard regression fix). --- ## 2026-07-09 — PRI-440/439: P11 hardening, pricing analytics, SDK 0.1.1 ### New features - **Brownout ladder + status page** — spend anomaly auto-suspend with configurable ladder (`BROWNOUT_STATE`). Public status reflected at `/v1/status`. - **SSRF IP-pinning** — all outbound URL fetches (e.g., source_url ingest) are blocked from internal RFC-1918 and link-local ranges. - **Key-hygiene redaction** — raw key values are redacted from server logs after mint. - **C1 runtime measurement** — per-analysis GPU wall-clock recorded for pricing analytics (Phase 5). ### SDK - **`PrimateError.rawMessage`** — verbatim server message, for UI display. TS SDK + MCP bumped to 0.1.1 via npm OIDC Trusted Publishing. - **Python SDK** — PyPI Trusted Publishing workflow added (tag `sdk-py-v*`). --- <!-- Backfilled 2026-07-27 (PRI-482 r8) from the website changelog when docs collapsed to one clock. --> ## 2026-07-09 — `additive` — SDKs, MCP server, agent artifacts - Official SDKs: TypeScript (`@primate-intelligence/sdk`) and Python (`primate-intelligence`) — typed resources, registry-driven retries, auto idempotency keys, `createAndWait()`, webhook verification, pagination iterators, browser `connectStream()`. - MCP server (`@primate-intelligence/mcp`) with seven tools mirroring the spec. - `llms.txt` / `llms-full.txt` + `/.well-known/llms.txt`; all docs pages served as raw markdown at `/docs/*.md`. - New documentation site: quickstart, agent quickstart, ten guides, Scalar API reference, error-registry anchors. ## 2026-07-09 — `deprecation` — Legacy `pk_` API key prefix sunset prepared - Existing `pk_` keys remain valid through the launch migration. - New keys and rotations create `pv_live_` / `pv_test_` keys only. - The 12-month `pk_` sunset clock starts at production GA announcement after Matt approves the cutover; the exact date will be published in the versioning guide and dashboard before enforcement. ## 2026-07-08 — `additive` — Billing: credit purchases + auto-refill - `GET /v1/credit-pricing` (public): price per second, grant sizes, purchase options. - Dashboard credit purchases, saved payment methods, and threshold-triggered auto-refill. - `GET /v1/billing/usage-summary` for dashboard usage aggregation. ## 2026-07-07 — `additive` — Webhooks + usage meters - `webhook_endpoints` resource: CRUD, Standard-Webhooks signing, delivery worker with `1m…24h` retry schedule, deliveries API, redelivery, secret rotation with 24h dual-secret overlap, auto-disable after 72h of failures. - Event vocabulary: `video.ready`, `video.failed`, `analysis.completed`, `analysis.failed`, `analysis.canceled`, `stream.ended`. - Per-request `webhook` overrides on `POST /v1/analyses` / `POST /v1/streams` (unsigned). - `GET /v1/usage` generic meters shape (`credit_seconds` balance + period consumption). ## 2026-07-06 — `additive` — Streams public API - `streams` resource (§4.8): create → signal (WS) → WebRTC → live per-frame results → end. Queueing with honest `queue_position`/`estimated_start_s`, per-plan session caps, 5s metering ticks, credit reservation reconciled to the second. - `stream_already_active` (409) error code registered. ## 2026-07-05 — `additive` — Client tokens - `POST /v1/client_tokens`: ephemeral browser-safe `pvct_` tokens (60–900s TTL, scope subsets, optional video/stream binding). Signaling WS accepts client tokens only — never secret keys. - `token_expired` (401) error code registered. ## 2026-07-04 — `additive` — Instant sandbox + auth cutoff - Anonymous `POST /v1/sandbox` issues instant `pv_test_` keys (IP-limited, 7-day expiry, fixture corpus, deterministic results) with a pre-seeded fixture video. - Metered credit enforcement on the live plane; open signup. - `sandbox_limit_exceeded` (429) error code registered. ## 2026-07-03 — `additive` — Public core: videos + analyses - `videos` resource: presigned upload (create → PUT → complete), SSRF-guarded URL ingest, cursor-paginated lists, delete with 409 protection. - `analyses` resource: free-text prompts or structured queries, `Prefer: wait` sync sugar (cap 120s), live progress + queue position, cancel, deterministic `result` contract (`answer`/`confidence`/`clips`). - `GET /v1/models`, `GET /v1/errors` (machine-readable registry), idempotency keys (JCS canonical body comparison, 24h replay window). ## 2026-07-02 — `additive` — OpenAPI spec - `GET /v1/openapi.json` — OpenAPI 3.1, generated from the DTO registry; CI drift + `oasdiff breaking` gates. The spec is the source of truth for docs, SDKs, and this changelog's compatibility promises. > Canonical, always-current copy: https://api.primateintelligence.ai/docs/changelog.md (also served at https://primateintelligence.ai/docs/changelog.md — same bytes, proxied).