> ## Documentation Index > Fetch the complete documentation index at: https://docs.vainona.ai/llms.txt > Use this file to discover all available pages before exploring further. # Get the calibration report for the active version > How well the active version's answers match the outcomes joined to its evaluations, per engine epoch, before and after calibration (§6.10). Calibration is fitted per judgment version and per engine epoch, first shortly after the judgment's first outcomes and then nightly. From 100 outcomes it is Platt scaling for `bool` and temperature scaling for `choice` and `score`; from 1,000, isotonic regression. An epoch also needs 20 outcomes of each class: 20 `true` and 20 `false` for a bool, and two values with 20 each for a choice or score. Until then it has no calibration, its `calibrated` metrics are null, and `not_fitted` says why and what to post. `coverage` and `warnings` say where the outcomes sit, and warn when most come from answers a person reviewed. The raw metrics are computed at request time and include every outcome posted so far. See [calibration](/concepts/calibration). On a prefix path the report pools the outcomes of every namespace under the prefix, which is the fit inherited answers use (§7.7). ## OpenAPI ````yaml /api-reference/openapi.yaml get /namespaces/{ns}/judgments/{name}/calibration openapi: 3.1.0 info: title: Vainona API version: '1' license: name: Proprietary identifier: LicenseRef-Proprietary description: | The `/v1` contract from main spec §6. It is locked: changes within `/v1` are additive only, and anything breaking is `/v2`. - **Base URL.** `https://api.vainona.ai/v1`. - **Evolution.** Response objects may gain fields and enums may gain values. Clients must ignore what they do not know. Request objects reject unknown fields. - **Path segments.** Namespace names (`{ns}`) and document ids (`{id}`) may contain `/`. Send it percent-encoded as `%2F`, so that `acme/prod/tenant_123` is `acme%2Fprod%2Ftenant_123`. - **Templates (v1.5).** The judgment routes also take a namespace prefix ending in `/*`, such as `acme%2Fprod%2F*`. A judgment created there is a template: every namespace under the prefix inherits it, including ones created later. The most specific matching prefix wins (§7.7). - **Thresholds are settings (v1.5).** Named thresholds belong to the judgment, not to a version. They are evaluated at read time against the raw fields, so a change applies at once to every answer, with no recompute and no new version (§6.5). - **Entity judgments (entities, E1).** A judgment can read the documents that point at the judged one, through `context.related`, and apply only to documents matching `applies_to` (§6.5.2). Its answers carry a `watermark`, and creating one that runs `on_change` returns a replay estimate of its monthly cost until you confirm (§6.9). - **Referenced documents (entities, E3a).** A relation can also read the one document the judged document points at, through `join: {theirs: "id", mine: "attributes."}`. A change to what the relation renders of that document re-judges the judged documents that point at it and are inside `freshness.fanout.scope`: a fan-out. A large fan-out is a `fanout` job that waits for confirm (§6.5.2, §6.9). - **Idempotency.** Every write operation is idempotent by construction. `Idempotency-Key` is accepted on every mutating route and, while the node still caches the original response, returns it verbatim. - **Errors.** Every error is the `Error` envelope; the HTTP status follows the code (see each response). - **Rate limits.** Every response carries `X-RateLimit-Limit` and `X-RateLimit-Remaining`; a `rate_limited` response adds `Retry-After`, except a daily allowance that is used up, which says when it renews in `details.resets_at` instead. - **Usage.** Every response that bills carries `usage`. Judging is billed in engine-neutral judgment units: one judgment answered, per started 1,000 tokens of its compiled context and its question together, at least 1 (§9). No response ever carries an engine's price. servers: - url: https://api.vainona.ai/v1 security: - apiKey: [] tags: - name: Namespaces description: Namespaces, their settings and cache warming (§6.2, §6.11). - name: Documents description: Writes, point reads and queries (§6.3 to §6.8). - name: Judgments description: >- Judgment definitions, versions, settings, activation, backfill and template detach (§6.5, §6.9, §7.7). - name: Outcomes and calibration description: | Post what actually happened to judged documents, and read how well answers match it: the calibration report, calibrated answers and threshold recommendations (§6.10). See [calibration](/concepts/calibration). - name: Jobs description: >- Backfill, shadow, periodic, deletion, reference index and fan-out jobs (§6.2, §6.9, §7.10.5). - name: Engines description: The engine registry (§6.5, §7.4.5). paths: /namespaces/{ns}/judgments/{name}/calibration: parameters: - $ref: '#/components/parameters/NamespaceOrTemplate' - $ref: '#/components/parameters/JudgmentName' get: tags: - Outcomes and calibration summary: Get the calibration report for the active version description: | How well the active version's answers match the outcomes joined to its evaluations, per engine epoch, before and after calibration (§6.10). Calibration is fitted per judgment version and per engine epoch, first shortly after the judgment's first outcomes and then nightly. From 100 outcomes it is Platt scaling for `bool` and temperature scaling for `choice` and `score`; from 1,000, isotonic regression. An epoch also needs 20 outcomes of each class: 20 `true` and 20 `false` for a bool, and two values with 20 each for a choice or score. Until then it has no calibration, its `calibrated` metrics are null, and `not_fitted` says why and what to post. `coverage` and `warnings` say where the outcomes sit, and warn when most come from answers a person reviewed. The raw metrics are computed at request time and include every outcome posted so far. See [calibration](/concepts/calibration). On a prefix path the report pools the outcomes of every namespace under the prefix, which is the fit inherited answers use (§7.7). operationId: getCalibrationReport responses: '200': description: The report. headers: X-RateLimit-Limit: $ref: '#/components/headers/RateLimitLimit' X-RateLimit-Remaining: $ref: '#/components/headers/RateLimitRemaining' content: application/json: schema: $ref: '#/components/schemas/CalibrationReport' '400': $ref: '#/components/responses/InvalidRequest' '401': $ref: '#/components/responses/Unauthorized' '403': $ref: '#/components/responses/Forbidden' '404': $ref: '#/components/responses/NotFound' '409': $ref: '#/components/responses/Conflict' '413': $ref: '#/components/responses/TooLarge' '429': $ref: '#/components/responses/RateLimited' '500': $ref: '#/components/responses/Internal' components: parameters: NamespaceOrTemplate: name: ns in: path required: true description: | A namespace name, or a template prefix ending in `/*` (§7.7), with any `/` sent as `%2F`: `acme%2Fprod%2Ftenant_123` or `acme%2Fprod%2F*`. schema: $ref: '#/components/schemas/NamespaceOrTemplate' JudgmentName: name: name in: path required: true schema: $ref: '#/components/schemas/Name' headers: RateLimitLimit: description: Requests allowed per second for this key. schema: type: integer minimum: 0 RateLimitRemaining: description: Requests left in the current window. schema: type: integer minimum: 0 RetryAfter: description: Seconds to wait before retrying. schema: type: integer minimum: 0 schemas: CalibrationReport: type: object description: | How well the active version's answers match their outcomes (§6.10). `epochs` holds one entry per engine epoch with outcomes, newest first. An epoch is the `engine_version` its evaluations were computed under: each Jev epoch label, or the exact version for a pinned engine. required: - judgment - type - version - outcomes - epochs properties: judgment: $ref: '#/components/schemas/Name' type: type: string enum: - bool - choice - score version: $ref: '#/components/schemas/JudgmentVersionNumber' description: The active version the report is for. outcomes: type: integer format: int64 minimum: 0 description: Outcomes joined to evaluations of this version, across every epoch. epochs: type: array items: $ref: '#/components/schemas/CalibrationEpoch' NamespaceOrTemplate: type: string description: | A namespace name, or a template prefix: a namespace path ending in `/*`, such as `acme/prod/*`, which every namespace under `acme/prod/` inherits judgments from (§7.7). Up to 256 bytes. pattern: ^[A-Za-z0-9._:/-]+(/\*)?$ minLength: 1 maxLength: 256 Name: type: string description: >- A judgment, attribute or threshold name. Names are path segments in field references, so they never contain `.`. pattern: ^[A-Za-z0-9_-]+$ minLength: 1 maxLength: 128 JudgmentVersionNumber: type: integer format: int32 minimum: 1 CalibrationEpoch: type: object description: | One epoch's metrics. For a composite judgment (§6.5.1), `raw` measures the combined `p` and nothing is calibrated: `method`, `fitted_at` and `calibrated` are null. required: - engine - engine_version - outcomes - implicit_negatives - method - fitted_at - from_previous_epoch - not_fitted - coverage - warnings - raw - calibrated properties: engine: type: string engine_version: type: string description: The epoch, as evaluations and answers record it. outcomes: type: integer format: int64 minimum: 0 description: >- Outcomes joined to this epoch's evaluations, implicit negatives included. implicit_negatives: type: integer format: int64 minimum: 0 description: >- How many of `outcomes` are implicit negatives (`outcomes.implicit_negatives` on the judgment). 0 when the setting is off. method: description: Null while nothing is fitted; `not_fitted` says why. oneOf: - $ref: '#/components/schemas/CalibrationMethod' - type: 'null' fitted_at: description: >- When the current fit was made (nightly). Null when nothing is fitted. oneOf: - $ref: '#/components/schemas/Timestamp' - type: 'null' from_previous_epoch: type: boolean description: | True when this epoch has no fit of its own yet (after a Jev drift) and its answers read with the previous epoch's calibration, or for a composite judgment the previous epoch's combiner. They are marked `from_previous_epoch` too. not_fitted: description: >- Why this epoch has no fit of its own. Null when it has one, and for a composite judgment. oneOf: - $ref: '#/components/schemas/CalibrationShortfall' - type: 'null' coverage: description: >- The lowest and highest raw value among this epoch's outcomes. Null when it has none. oneOf: - $ref: '#/components/schemas/RawRange' - type: 'null' warnings: type: array description: >- Where the outcomes sit that makes calibration less reliable. Empty when there is nothing to say. items: $ref: '#/components/schemas/CoverageWarning' raw: description: The metrics of the raw answers. Null when the epoch has no outcomes. oneOf: - $ref: '#/components/schemas/CalibrationMetrics' - type: 'null' calibrated: description: The same metrics after calibration. Null when nothing is fitted. oneOf: - $ref: '#/components/schemas/CalibrationMetrics' - type: 'null' Error: type: object required: - error properties: error: type: object required: - code - message properties: code: $ref: '#/components/schemas/ErrorCode' message: type: string details: type: object description: >- Code-specific detail, such as the scan estimate on a refused query. Every `internal` error has `request_id`. CalibrationMethod: type: string description: >- `platt`: from 100 outcomes, Platt scaling for `bool` and temperature scaling over the distribution for `choice` and `score`. `isotonic`: isotonic regression, from 1,000. enum: - platt - isotonic Timestamp: type: string format: date-time description: RFC 3339, UTC. CalibrationShortfall: type: object description: | Why there is no calibration or no threshold recommendation (§6.10). `message` says what to post, such as "Only positive outcomes so far: post outcomes for documents where it did not happen." when every outcome is `true`. required: - reason - message properties: reason: type: string description: | - `too_few_outcomes`: fewer than 100. - `too_few_positives`, `too_few_negatives`: fewer than 20 `true` or `false` outcomes (bool), or for a choice option, fewer than 20 outcomes of it or of the other options. - `too_few_values`: a choice or score has fewer than 2 values with 20 outcomes each. - `awaiting_fit`: the report only. There are enough outcomes, and the next fit, within a day, calibrates the epoch. enum: - too_few_outcomes - too_few_positives - too_few_negatives - too_few_values - awaiting_fit message: type: string RawRange: type: object description: >- A range of raw values, `p` or the probability of the most probable option or level. required: - lower - upper properties: lower: $ref: '#/components/schemas/Probability' upper: $ref: '#/components/schemas/Probability' CoverageWarning: type: object description: | Outcomes bunched where a reviewed sample puts them (§6.10), so the fit is reliable only there. - `above_thresholds`: more than 80% of the outcomes are on answers that meet one of the judgment's thresholds. - `high_band`: a bool judgment only. More than 80% of the outcomes are on answers with `p` of 0.7 or more. required: - code - message properties: code: type: string enum: - above_thresholds - high_band message: type: string CalibrationMetrics: type: object description: | Metrics over the joined outcomes. For choice and score judgments, confidence is the probability of the most probable option or level, and a hit is an outcome equal to it (the exact level for scores). required: - accuracy - expected_calibration_error - log_loss - reliability properties: accuracy: $ref: '#/components/schemas/Probability' description: >- Share of hits: `p` of at least 0.5 on a `true` outcome or below it on `false` (bool), the exact option or level (choice, score). expected_calibration_error: type: number minimum: 0 description: >- The count-weighted mean gap between `mean_predicted` and `observed` over the reliability bins. log_loss: type: number minimum: 0 description: Mean negative log probability given to the observed outcome. mean_level_distance: type: number minimum: 0 description: >- Score judgments only. Mean absolute distance, in levels, between the most probable level and the observed one. reliability: type: array description: >- The reliability curve, ten equal-width bins of predicted probability from 0 to 1, lowest first. minItems: 10 maxItems: 10 items: $ref: '#/components/schemas/ReliabilityBin' ErrorCode: type: string description: | HTTP status by code: `invalid_request` 400, `unauthorized` 401, `budget_exceeded` 402, `forbidden` 403, `not_found` 404, `conflict` 409, `too_large` 413, `engine_version_unavailable` 422, `insufficient_labels` 422, `rate_limited` 429, `internal` 500, `engine_unavailable` 503. enum: - invalid_request - unauthorized - forbidden - not_found - conflict - too_large - rate_limited - budget_exceeded - engine_unavailable - engine_version_unavailable - insufficient_labels - internal Probability: type: number minimum: 0 maximum: 1 ReliabilityBin: type: object description: >- Predictions whose probability (`p`, or the confidence for choice and score) falls in `[lower, upper)`; the last bin includes 1. required: - lower - upper - count - mean_predicted - observed properties: lower: $ref: '#/components/schemas/Probability' upper: $ref: '#/components/schemas/Probability' count: type: integer format: int64 minimum: 0 mean_predicted: description: Mean predicted probability in the bin. Null when the bin is empty. oneOf: - $ref: '#/components/schemas/Probability' - type: 'null' observed: description: >- Share of `true` outcomes (bool) or hits (choice, score) in the bin. Null when the bin is empty. oneOf: - $ref: '#/components/schemas/Probability' - type: 'null' responses: InvalidRequest: description: '`invalid_request`: the request is malformed or breaks a rule of §6.' content: application/json: schema: $ref: '#/components/schemas/Error' Unauthorized: description: '`unauthorized`: the key is missing, unknown or revoked.' content: application/json: schema: $ref: '#/components/schemas/Error' Forbidden: description: '`forbidden`: the key''s role or namespace prefix does not allow this.' content: application/json: schema: $ref: '#/components/schemas/Error' NotFound: description: '`not_found`: the namespace, document, judgment or job does not exist.' content: application/json: schema: $ref: '#/components/schemas/Error' Conflict: description: >- `conflict`: the namespace is being deleted, an object a read needs was compacted away and the read (or the query, without its cursor) must restart (§7.5.11), or the judgment is inherited from a template and this namespace cannot create, activate or delete it (§7.7). content: application/json: schema: $ref: '#/components/schemas/Error' TooLarge: description: >- `too_large`: the request body (on any route), a document, or the estimated query scan is over its limit. `details` carries the estimate for queries. It also answers a write that would take `default/quickstart` past its size cap (10,000 documents or 100 MB), with `details.max_documents` and `details.max_bytes`. content: application/json: schema: $ref: '#/components/schemas/Error' RateLimited: description: >- `rate_limited`: slow down. `Retry-After` says for how long, except when a daily allowance is used up: then `details.resets_at` says when it renews, and there is no `Retry-After`. headers: X-RateLimit-Limit: $ref: '#/components/headers/RateLimitLimit' X-RateLimit-Remaining: $ref: '#/components/headers/RateLimitRemaining' Retry-After: $ref: '#/components/headers/RetryAfter' content: application/json: schema: $ref: '#/components/schemas/Error' Internal: description: >- `internal`: something failed on our side. Nothing partial is returned, and a request never commits part of itself. Most `internal` errors committed nothing. A write can instead fail with a message saying it may have been committed, when we could not tell whether its batch landed (§7.2.10): it then committed all of it or none of it. Retrying a write is safe either way, because writes are idempotent. The message is generic and carries a request id, also in `details.request_id`, which support uses to find the cause. content: application/json: schema: $ref: '#/components/schemas/Error' securitySchemes: apiKey: type: http scheme: bearer description: | An organization API key. Keys carry a role (`read_write` or `read_only`) and may be restricted to a namespace prefix such as `acme/*`, or to one namespace such as `acme/prod/tenant_1`. A prefix matches on a `/` boundary: `acme/prod/tenant_1*` covers `acme/prod/tenant_1` and everything under `acme/prod/tenant_1/`, never `acme/prod/tenant_12`. ````