> ## Documentation Index > Fetch the complete documentation index at: https://docs.vainona.ai/llms.txt > Use this file to discover all available pages before exploring further. # Make a version the one new evaluations use > Existing answers keep their `judgment_version` until recomputed. Activating a version whose engine or definition differs from the active one without `force` returns a `shadow` job in `awaiting_confirm` (`202`). The job evaluates a random sample of 1,000 documents under the new version, never producing answers, and its `report` compares old and new (§7.3.11). `POST /jobs/{id}/confirm` completes the switch, at any point; cancelling the job leaves the active version as it was. With `force: true`, or when nothing differs, the version is active at once (`200`). On a prefix path the sample is drawn across every namespace under the prefix, and confirming switches them all. A namespace cannot activate a version of a judgment it inherits (`conflict`): it follows the template's active version (§7.7). Activating a composite version (§6.5.1) always returns the shadow job, even with `force`. Its sample is up to 1,000 documents with outcomes for this judgment; it fits the combiner with 5-fold cross-validation and compares it with the fair baseline (`CompositeShadowReport`). Fewer than 50 labelled documents, or fewer than 10 of either answer, fail the job with `insufficient_labels`. `confirm` activates the version with the combiner fitted on all of them, whatever the `verdict`. ## OpenAPI ````yaml /api-reference/openapi.yaml post /namespaces/{ns}/judgments/{name}/activate openapi: 3.1.0 info: title: Vainona API version: '1' license: name: Proprietary identifier: LicenseRef-Proprietary description: | The `/v1` contract from main spec §6. It is locked: changes within `/v1` are additive only, and anything breaking is `/v2`. - **Base URL.** `https://api.vainona.ai/v1`. - **Evolution.** Response objects may gain fields and enums may gain values. Clients must ignore what they do not know. Request objects reject unknown fields. - **Path segments.** Namespace names (`{ns}`) and document ids (`{id}`) may contain `/`. Send it percent-encoded as `%2F`, so that `acme/prod/tenant_123` is `acme%2Fprod%2Ftenant_123`. - **Templates (v1.5).** The judgment routes also take a namespace prefix ending in `/*`, such as `acme%2Fprod%2F*`. A judgment created there is a template: every namespace under the prefix inherits it, including ones created later. The most specific matching prefix wins (§7.7). - **Thresholds are settings (v1.5).** Named thresholds belong to the judgment, not to a version. They are evaluated at read time against the raw fields, so a change applies at once to every answer, with no recompute and no new version (§6.5). - **Entity judgments (entities, E1).** A judgment can read the documents that point at the judged one, through `context.related`, and apply only to documents matching `applies_to` (§6.5.2). Its answers carry a `watermark`, and creating one that runs `on_change` returns a replay estimate of its monthly cost until you confirm (§6.9). - **Referenced documents (entities, E3a).** A relation can also read the one document the judged document points at, through `join: {theirs: "id", mine: "attributes."}`. A change to what the relation renders of that document re-judges the judged documents that point at it and are inside `freshness.fanout.scope`: a fan-out. A large fan-out is a `fanout` job that waits for confirm (§6.5.2, §6.9). - **Idempotency.** Every write operation is idempotent by construction. `Idempotency-Key` is accepted on every mutating route and, while the node still caches the original response, returns it verbatim. - **Errors.** Every error is the `Error` envelope; the HTTP status follows the code (see each response). - **Rate limits.** Every response carries `X-RateLimit-Limit` and `X-RateLimit-Remaining`; a `rate_limited` response adds `Retry-After`, except a daily allowance that is used up, which says when it renews in `details.resets_at` instead. - **Usage.** Every response that bills carries `usage`. Judging is billed in engine-neutral judgment units: one judgment answered, per started 1,000 tokens of its compiled context and its question together, at least 1 (§9). No response ever carries an engine's price. servers: - url: https://api.vainona.ai/v1 security: - apiKey: [] tags: - name: Namespaces description: Namespaces, their settings and cache warming (§6.2, §6.11). - name: Documents description: Writes, point reads and queries (§6.3 to §6.8). - name: Judgments description: >- Judgment definitions, versions, settings, activation, backfill and template detach (§6.5, §6.9, §7.7). - name: Outcomes and calibration description: | Post what actually happened to judged documents, and read how well answers match it: the calibration report, calibrated answers and threshold recommendations (§6.10). See [calibration](/concepts/calibration). - name: Jobs description: >- Backfill, shadow, periodic, deletion, reference index and fan-out jobs (§6.2, §6.9, §7.10.5). - name: Engines description: The engine registry (§6.5, §7.4.5). paths: /namespaces/{ns}/judgments/{name}/activate: parameters: - $ref: '#/components/parameters/NamespaceOrTemplate' - $ref: '#/components/parameters/JudgmentName' post: tags: - Judgments summary: Make a version the one new evaluations use description: | Existing answers keep their `judgment_version` until recomputed. Activating a version whose engine or definition differs from the active one without `force` returns a `shadow` job in `awaiting_confirm` (`202`). The job evaluates a random sample of 1,000 documents under the new version, never producing answers, and its `report` compares old and new (§7.3.11). `POST /jobs/{id}/confirm` completes the switch, at any point; cancelling the job leaves the active version as it was. With `force: true`, or when nothing differs, the version is active at once (`200`). On a prefix path the sample is drawn across every namespace under the prefix, and confirming switches them all. A namespace cannot activate a version of a judgment it inherits (`conflict`): it follows the template's active version (§7.7). Activating a composite version (§6.5.1) always returns the shadow job, even with `force`. Its sample is up to 1,000 documents with outcomes for this judgment; it fits the combiner with 5-fold cross-validation and compares it with the fair baseline (`CompositeShadowReport`). Fewer than 50 labelled documents, or fewer than 10 of either answer, fail the job with `insufficient_labels`. `confirm` activates the version with the combiner fitted on all of them, whatever the `verdict`. operationId: activateJudgment parameters: - $ref: '#/components/parameters/IdempotencyKey' requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/ActivateJudgmentRequest' responses: '200': description: The version is active. headers: X-RateLimit-Limit: $ref: '#/components/headers/RateLimitLimit' X-RateLimit-Remaining: $ref: '#/components/headers/RateLimitRemaining' content: application/json: schema: $ref: '#/components/schemas/Judgment' '202': description: >- A shadow job was created and awaits confirmation. Its `report` fills in when the sample is done. headers: X-RateLimit-Limit: $ref: '#/components/headers/RateLimitLimit' X-RateLimit-Remaining: $ref: '#/components/headers/RateLimitRemaining' content: application/json: schema: $ref: '#/components/schemas/Job' '400': $ref: '#/components/responses/InvalidRequest' '401': $ref: '#/components/responses/Unauthorized' '403': $ref: '#/components/responses/Forbidden' '404': $ref: '#/components/responses/NotFound' '409': $ref: '#/components/responses/Conflict' '413': $ref: '#/components/responses/TooLarge' '422': $ref: '#/components/responses/EngineVersionUnavailable' '429': $ref: '#/components/responses/RateLimited' '500': $ref: '#/components/responses/Internal' components: parameters: NamespaceOrTemplate: name: ns in: path required: true description: | A namespace name, or a template prefix ending in `/*` (§7.7), with any `/` sent as `%2F`: `acme%2Fprod%2Ftenant_123` or `acme%2Fprod%2F*`. schema: $ref: '#/components/schemas/NamespaceOrTemplate' JudgmentName: name: name in: path required: true schema: $ref: '#/components/schemas/Name' IdempotencyKey: name: Idempotency-Key in: header description: >- Returns the original response verbatim while the node still caches it. Correctness never depends on it. schema: type: string minLength: 1 schemas: ActivateJudgmentRequest: type: object additionalProperties: false required: - version properties: version: $ref: '#/components/schemas/JudgmentVersionNumber' force: type: boolean default: false description: >- Activate at once, without a shadow report, when the engine or definition differs from the active version. Judgment: type: object required: - name - active_version - freshness - thresholds - outcomes - versions properties: name: $ref: '#/components/schemas/Name' active_version: description: Null once the judgment is deactivated. oneOf: - $ref: '#/components/schemas/JudgmentVersionNumber' - type: 'null' freshness: $ref: '#/components/schemas/FreshnessSettings' thresholds: $ref: '#/components/schemas/ThresholdSettings' outcomes: $ref: '#/components/schemas/OutcomeSettings' template: $ref: '#/components/schemas/NamespaceOrTemplate' description: >- Present when the judgment is inherited, the template prefix it comes from, such as `acme/prod/*` (§7.7). overrides: type: array description: >- Present when the judgment is inherited. The settings this namespace overrides; the others follow the template (§7.7). uniqueItems: true items: type: string enum: - thresholds - freshness fanout_sources: $ref: '#/components/schemas/FanoutSources' versions: type: array description: Every version, oldest first. Versions are retained forever. items: $ref: '#/components/schemas/JudgmentVersion' Job: type: object description: | A job row (§7.10.5). A `shadow` job comes from activating a differing version without `force` (§6.9). It starts in `awaiting_confirm` and runs its sample while it waits; `report` fills in when the sample is done. `confirm` activates `version` and the job ends `done`; `cancel` leaves the active version unchanged. On a template prefix, `namespace` is the prefix. A `reference_index` job (entities, E1) builds the index an entity judgment's relations need (§6.5.2); it starts `running`, and `attribute` names the attribute it indexes. A `fanout` job (entities, E3a) re-judges the judged documents one referenced change touches (§6.9): `referenced` names the change and `estimate` its cost. It waits in `awaiting_confirm` unless the judgment's `fanout.confirm_jobs` is `false`. `confirm` is `budget_exceeded` when the estimate does not fit what is left of the namespace's budget, and `cancel` leaves the in-scope answers `stale`. required: - id - type - status - namespace - spend_usd - progress - created_at - updated_at properties: id: type: string type: type: string enum: - backfill - shadow - periodic - namespace_delete - reference_index - fanout status: type: string enum: - estimating - awaiting_confirm - running - paused - done - failed - cancelled namespace: $ref: '#/components/schemas/NamespaceOrTemplate' judgment: $ref: '#/components/schemas/Name' estimate: oneOf: - $ref: '#/components/schemas/Estimate' - type: 'null' spend_usd: type: number minimum: 0 description: Judging billed by this job so far, in US dollars. progress: type: object required: - documents_done properties: documents_done: type: integer format: int64 minimum: 0 estimated_completion_at: oneOf: - $ref: '#/components/schemas/Timestamp' - type: 'null' error: type: - string - 'null' attribute: type: - string - 'null' pattern: ^attributes\.[A-Za-z0-9_-]+$ description: >- Entities, E1 (§6.5.2). `reference_index` jobs only, as `attributes.`. The attribute whose reference index the job builds. Null for other jobs. referenced: description: Entities, E3a (§6.9). `fanout` jobs only; null for other jobs. oneOf: - $ref: '#/components/schemas/ReferencedChange' - type: 'null' version: $ref: '#/components/schemas/JudgmentVersionNumber' description: Shadow jobs only. The version being activated. report: description: | Shadow jobs only. Null while the sample runs; `progress` counts the sampled documents. A composite version's job has a `CompositeShadowReport` (§6.5.1), and when it has too few labels it fails with an `error` that starts with `insufficient_labels`. oneOf: - $ref: '#/components/schemas/ShadowReport' - $ref: '#/components/schemas/CompositeShadowReport' - type: 'null' created_at: $ref: '#/components/schemas/Timestamp' updated_at: $ref: '#/components/schemas/Timestamp' NamespaceOrTemplate: type: string description: | A namespace name, or a template prefix: a namespace path ending in `/*`, such as `acme/prod/*`, which every namespace under `acme/prod/` inherits judgments from (§7.7). Up to 256 bytes. pattern: ^[A-Za-z0-9._:/-]+(/\*)?$ minLength: 1 maxLength: 256 Name: type: string description: >- A judgment, attribute or threshold name. Names are path segments in field references, so they never contain `.`. pattern: ^[A-Za-z0-9_-]+$ minLength: 1 maxLength: 128 JudgmentVersionNumber: type: integer format: int32 minimum: 1 FreshnessSettings: type: object description: Settings, not part of the definition. Changing them creates no version. additionalProperties: false properties: policy: $ref: '#/components/schemas/FreshnessPolicy' debounce_ms: type: integer minimum: 0 default: 0 description: >- A document changed more recently than this is not judged until it settles. interval: $ref: '#/components/schemas/Duration' description: >- Recompute interval for `periodic`, such as `1h` or `1d`. At least `1h`, because each run re-judges the whole namespace (§7.3.12); a shorter one is `invalid_request`. max_wait_ms: type: - integer - 'null' minimum: 1 description: | Entities, E1 (§7.3.9). The debounce ceiling: a document that keeps changing is still judged once this long has passed since its oldest unjudged change. At least `debounce_ms`; `null` means no ceiling. Defaults to 12 × `debounce_ms` for an entity judgment with a `debounce_ms` above 0, and to no ceiling otherwise: with no debounce a document is judged at once. `GET` returns the value in effect. fanout: $ref: '#/components/schemas/FanoutSettings' if: required: - policy properties: policy: const: periodic then: required: - interval ThresholdSettings: type: object description: | The judgment's named thresholds, a setting rather than part of a version (§6.5). Each is a number for a bool judgment (true when `p` is at least it) or a score judgment (true when `score` is at least it), and `{value, gte}` for a choice judgment (true when `dist[value]` is at least `gte`). They are evaluated at read time against the raw fields, never the `calibrated` ones. A threshold that does not fit the judgment's type is `invalid_request`. propertyNames: $ref: '#/components/schemas/Name' additionalProperties: oneOf: - type: number - $ref: '#/components/schemas/ChoiceThreshold' OutcomeSettings: type: object description: | How the judgment's outcomes are read (§6.10), a setting rather than part of a version. See [calibration](/concepts/calibration). additionalProperties: false properties: implicit_negatives: type: boolean default: false description: | For a `bool` judgment with a non-zero `horizon` only. When true, a judged document with no outcome counts as a `false` outcome once the horizon has passed since the revision it judged, so you only need to post what happened. A document with any outcome of its own gets no implicit one, and an evaluation whose horizon has not passed never counts. Leave it off when a missing outcome does not mean "no", for example when you only record some of what happens. On a namespace that inherits the judgment, set it on the template's prefix path: its calibration is shared. FanoutSources: type: object description: | Entities, E3a (§6.5.2). Where each value in `freshness.fanout` comes from, key for key. Present exactly when `freshness.fanout` is. A `where` removed with `null` is the default, no filter; any other `null` the judgment set, such as `created_within: null`, is the judgment's. required: - scope - debounce_ms - max_wait_ms - job_above - confirm_jobs - auto_daily_limit properties: scope: type: object required: - created_within - where properties: created_within: $ref: '#/components/schemas/FanoutSource' where: $ref: '#/components/schemas/FanoutSource' debounce_ms: $ref: '#/components/schemas/FanoutSource' max_wait_ms: $ref: '#/components/schemas/FanoutSource' job_above: $ref: '#/components/schemas/FanoutSource' confirm_jobs: $ref: '#/components/schemas/FanoutSource' auto_daily_limit: $ref: '#/components/schemas/FanoutSource' JudgmentVersion: type: object required: - version - created_at - definition properties: version: $ref: '#/components/schemas/JudgmentVersionNumber' created_at: $ref: '#/components/schemas/Timestamp' definition: $ref: '#/components/schemas/JudgmentDefinition' Estimate: type: object required: - documents - tokens - judgment_units - cost_usd - duration_s properties: documents: type: integer format: int64 minimum: 0 tokens: type: integer format: int64 minimum: 0 judgment_units: type: integer format: int64 minimum: 0 description: >- Estimated judgment units (§9), from a sample of up to 1,000 documents. cost_usd: type: number minimum: 0 description: >- What the backfill is expected to cost you, the judgment units at the graduated prices of the tiers your organization will be in, counting what it has already used this month (§9). duration_s: type: integer format: int64 minimum: 0 description: >- Expected wall-clock seconds at the bulk pool's current share of the engine's rate limit. Timestamp: type: string format: date-time description: RFC 3339, UTC. ReferencedChange: type: object description: | Entities, E3a (§6.9). The change a `fanout` job fans out: a write to the referenced document that changed what the relation renders. While the job waits for confirm, a later change to the same document moves `revision` to it, and one confirm covers them all. required: - relation - document_id - revision properties: relation: $ref: '#/components/schemas/Name' document_id: $ref: '#/components/schemas/DocumentId' revision: $ref: '#/components/schemas/Revision' description: >- The referenced document's revision at the change. Every judged document the job re-judges gets an answer with a `watermark` at or above it. ShadowReport: type: object description: | What activating `version` would change, from a shadow sample of 1,000 random documents, or every document when there are fewer (§7.3.11). Shadow evaluations are in the evaluation log with `shadow: true` and never produce answers. `current` is the answers of the active version for the sampled documents, and `candidate` the shadow evaluations. required: - documents - current - candidate - threshold_flips - recompute properties: documents: type: integer format: int64 minimum: 0 description: Documents in the sample. current: $ref: '#/components/schemas/ShadowSide' candidate: $ref: '#/components/schemas/ShadowSide' threshold_flips: type: object description: | Per named threshold, the sampled documents whose threshold boolean would change. The candidate side uses the thresholds that apply once the version is active. propertyNames: $ref: '#/components/schemas/Name' additionalProperties: $ref: '#/components/schemas/ThresholdFlips' recompute: $ref: '#/components/schemas/Estimate' description: >- What a backfill of every document under the new version would cost, as a backfill estimate. CompositeShadowReport: type: object description: | The shadow report of a composite version (§6.5.1). The job judged up to 1,000 documents with outcomes for this judgment and fitted the combiner with 5-fold cross-validation, so every document is scored by a fit that never saw its label. The baseline gets the same treatment: its cut-off is fitted by the same folds on the same outcomes, never the raw 0.5, so splitting gets no credit for what a fitted threshold alone would give. `verdict` is `better` only when the composite's accuracy beats the baseline's by more than the interval of the difference. Otherwise it is `not_better`, and the threshold recommender on the baseline is the cheaper fix. `confirm` activates the version either way, with the combiner fitted on all the documents. required: - type - documents - labels - composite - baseline - difference - parts - verdict - recompute properties: type: const: composite documents: type: integer format: int64 minimum: 0 description: Labelled documents judged and scored. labels: type: object description: The documents by their outcome. required: - 'true' - 'false' properties: 'true': type: integer format: int64 minimum: 0 'false': type: integer format: int64 minimum: 0 composite: $ref: '#/components/schemas/CrossValidated' baseline: $ref: '#/components/schemas/Baseline' difference: type: object description: >- The composite's cross-validated accuracy minus the baseline's, with its 95% interval over the documents (paired). required: - accuracy - interval properties: accuracy: type: number minimum: -1 maximum: 1 interval: $ref: '#/components/schemas/DifferenceInterval' parts: type: array description: >- Each part's weight in the combiner fitted on all the documents, on standardised log-odds, so sizes compare. A negative weight means a yes points to `false`. items: $ref: '#/components/schemas/PartWeight' features: type: array description: >- Entities, E1 (§6.5.1). Each feature's weight in the same fit, on the same standardised scale as the parts'. items: $ref: '#/components/schemas/FeatureWeight' verdict: type: string enum: - better - not_better recompute: $ref: '#/components/schemas/Estimate' description: >- What a backfill of every document under the new version would cost, as a backfill estimate. Each part is billed as a judgment. Error: type: object required: - error properties: error: type: object required: - code - message properties: code: $ref: '#/components/schemas/ErrorCode' message: type: string details: type: object description: >- Code-specific detail, such as the scan estimate on a refused query. Every `internal` error has `request_id`. FreshnessPolicy: type: string description: | `on_read` (default) computes on first read, then caches until the document changes. `on_change` is required for any judgment used in a query filter or rank. `periodic` recomputes every `interval`. `manual` is backfill only. enum: - on_read - on_change - periodic - manual Duration: type: string description: A whole number and a unit (`s`, `m`, `h` or `d`), such as `7d` or `1h`. pattern: ^[1-9][0-9]*[smhd]$ FanoutSettings: type: object description: | Entities, E3a (§6.5.2). For a judgment with a relation that reads a referenced document: which judged documents a change to it re-judges, and when. Only a change to what the relation renders counts. Every key is optional in a request, and `GET` returns every value in effect for a judgment with such a relation. In a `PATCH`, the keys you send replace those keys and the others keep their values. additionalProperties: false minProperties: 1 properties: scope: $ref: '#/components/schemas/FanoutScope' debounce_ms: type: integer minimum: 0 default: 600000 description: >- A referenced document that changed more recently than this is not fanned out until it settles, so a burst of changes to it costs one fan-out. Counts changes to what the relation renders, per referenced document. The judgment's own `debounce_ms` still governs the judged documents' own writes. max_wait_ms: type: - integer - 'null' minimum: 1 description: >- The ceiling. A referenced document that keeps changing still fans out once this long has passed since its oldest change not yet fanned out. At least `debounce_ms`; `null` means no ceiling. Defaults to 6 × `debounce_ms`, one hour at the default debounce. job_above: type: - integer - 'null' format: int64 minimum: 0 description: >- A fan-out that would re-judge more than this many judged documents is a visible `fanout` job with an estimate (§6.9). `0` makes every fan-out a job. Defaults to the namespace's `fanout.job_above`, 10,000 unless it sets one. `null` in a request clears the judgment's own value, so it follows the namespace's again; `GET` never returns `null`. confirm_jobs: type: - boolean - 'null' description: >- A `fanout` job waits in `awaiting_confirm` until `POST /jobs/{id}/confirm`. `false` starts it at once, unless its estimate does not fit what is left of the namespace's budget. Defaults to the namespace's `fanout.confirm_jobs`, `true` unless it sets one. `null` in a request clears the judgment's own value, so it follows the namespace's again; `GET` never returns `null`. auto_daily_limit: type: - integer - 'null' format: int64 minimum: 0 default: 100000 description: >- The most judged documents fan-outs of this judgment re-judge automatically in any 24 hours. Once a fan-out would pass it, it becomes a `fanout` job that waits for confirm, however small, as one above `job_above` does. `null` means no limit. ChoiceThreshold: type: object additionalProperties: false required: - value - gte properties: value: type: string minLength: 1 gte: $ref: '#/components/schemas/Probability' FanoutSource: type: string description: | Entities, E3a (§6.5.2). Where a fan-out value in effect comes from: `judgment` when the judgment set it (on an inherited judgment, its template or this namespace's override), `namespace` when the judgment follows the namespace's `fanout` (`job_above` and `confirm_jobs` only), and `default` for the built-in default. enum: - judgment - namespace - default JudgmentDefinition: type: object description: | A judgment definition, discriminated on `type`. `thresholds` are the named thresholds given with this version. They are not part of the version: the judgment's current thresholds are its `thresholds` setting, evaluated at read time and returned as booleans under `answers.{name}.thresholds` (§6.5). oneOf: - $ref: '#/components/schemas/BoolJudgmentDefinition' - $ref: '#/components/schemas/ChoiceJudgmentDefinition' - $ref: '#/components/schemas/ScoreJudgmentDefinition' discriminator: propertyName: type mapping: bool: $ref: '#/components/schemas/BoolJudgmentDefinition' choice: $ref: '#/components/schemas/ChoiceJudgmentDefinition' score: $ref: '#/components/schemas/ScoreJudgmentDefinition' DocumentId: type: string description: Up to 128 bytes. pattern: ^[A-Za-z0-9._:/-]+$ minLength: 1 maxLength: 128 Revision: type: integer format: int64 minimum: 0 ShadowSide: type: object description: One side of a shadow report, discriminated on the judgment `type`. oneOf: - $ref: '#/components/schemas/ShadowValues' - $ref: '#/components/schemas/ShadowOptions' discriminator: propertyName: type mapping: bool: $ref: '#/components/schemas/ShadowValues' score: $ref: '#/components/schemas/ShadowValues' choice: $ref: '#/components/schemas/ShadowOptions' ThresholdFlips: type: object required: - to_true - to_false properties: to_true: type: integer format: int64 minimum: 0 description: Documents false under the active version and true under the new one. to_false: type: integer format: int64 minimum: 0 description: Documents true under the active version and false under the new one. CrossValidated: type: object description: Out-of-fold scores over the report's documents. required: - accuracy - accuracy_interval - auc properties: accuracy: $ref: '#/components/schemas/Probability' description: >- Share of documents whose out-of-fold probability is on the right side of 0.5. accuracy_interval: $ref: '#/components/schemas/MetricInterval' auc: $ref: '#/components/schemas/Probability' description: ROC AUC of the out-of-fold probabilities. Baseline: type: object description: >- The fair baseline, scored like the composite. The active version (`version`), or for a first version the composite's `question` asked on its own (`version` null). required: - version - threshold - accuracy - accuracy_interval - auc allOf: - $ref: '#/components/schemas/CrossValidated' properties: version: oneOf: - $ref: '#/components/schemas/JudgmentVersionNumber' - type: 'null' threshold: description: >- The baseline's fitted cut-off on its `p`, fitted on all the documents. Null when a higher `p` does not mean `true` more often. oneOf: - $ref: '#/components/schemas/Probability' - type: 'null' DifferenceInterval: type: object required: - lower - upper properties: lower: type: number minimum: -1 maximum: 1 upper: type: number minimum: -1 maximum: 1 PartWeight: type: object required: - name - weight properties: name: $ref: '#/components/schemas/Name' weight: type: number FeatureWeight: type: object required: - name - weight properties: name: $ref: '#/components/schemas/FeatureRef' weight: type: number ErrorCode: type: string description: | HTTP status by code: `invalid_request` 400, `unauthorized` 401, `budget_exceeded` 402, `forbidden` 403, `not_found` 404, `conflict` 409, `too_large` 413, `engine_version_unavailable` 422, `insufficient_labels` 422, `rate_limited` 429, `internal` 500, `engine_unavailable` 503. enum: - invalid_request - unauthorized - forbidden - not_found - conflict - too_large - rate_limited - budget_exceeded - engine_unavailable - engine_version_unavailable - insufficient_labels - internal FanoutScope: type: object description: | Entities, E3a (§6.5.2). The re-judge scope: which judged documents a change to a referenced document at log position c re-judges. At the fan-out's snapshot, a judged document is in scope when it points at the referenced document, matches `applies_to` and `where`, and was created no more than `created_within` before the write at c. A judged document outside the scope keeps its answer, which is not marked `pending`; its `watermark` says which version of the referenced document it read. additionalProperties: false minProperties: 1 properties: created_within: description: >- Only judged documents created no more than this long before the change. `null` means all of them. Defaults to `30d`. oneOf: - $ref: '#/components/schemas/Duration' - type: 'null' where: description: >- Narrows the scope to judged documents whose attributes match, such as `{"attributes.status": "open"}`. `null` removes it. None by default. oneOf: - $ref: '#/components/schemas/AttributeFilter' - type: 'null' Probability: type: number minimum: 0 maximum: 1 BoolJudgmentDefinition: type: object description: | The answer has `p`. With `parts` it is a composite judgment (§6.5.1): the engine is asked each part instead of `question`, and `p` combines the parts' answers with weights fitted on the judgment's outcomes. required: - type allOf: - $ref: '#/components/schemas/JudgmentDefinitionCommon' dependentRequired: features: - parts not: description: A single part needs features. required: - parts properties: parts: type: array maxItems: 1 not: required: - features properties: type: const: bool thresholds: type: object description: Each threshold is true when `p` is at least the value. propertyNames: $ref: '#/components/schemas/Name' additionalProperties: $ref: '#/components/schemas/Probability' parts: type: array description: | Composite judgments only: 2 to 8 narrow yes/no questions with names unique within the judgment. They share the judgment's context recipe and engine, so they go in one engine request with its other questions, and each counts toward the 32 questions per request. `question` then documents what the combination means and is not sent to the engine. Parts are part of the version. They cannot reference other judgments. Each part is billed as a judgment (§9). With `features`, one part is enough. minItems: 1 maxItems: 8 items: $ref: '#/components/schemas/JudgmentPart' features: type: array description: | Entities, E1 (§6.5.1). Composite judgments only: aggregates over related documents that the combiner takes as numeric inputs beside the parts. Each names an aggregate the recipe's `related` declares. The combiner takes sign(x) × ln(1 + |x|) of each value and standardises it with the parts' log-odds; a missing value counts as the feature's mean. Features add no questions, so they add no judgment units. minItems: 1 maxItems: 8 uniqueItems: true items: $ref: '#/components/schemas/FeatureRef' ChoiceJudgmentDefinition: type: object description: | The answer has `value`, `dist` and `escape_p`. The API appends `none_of_the_above` as a final option on every choice judgment. required: - type - options allOf: - $ref: '#/components/schemas/JudgmentDefinitionCommon' properties: type: const: choice options: type: array description: Option values must be distinct. minItems: 2 maxItems: 254 items: $ref: '#/components/schemas/ChoiceOption' thresholds: type: object description: Each threshold is true when `dist[value]` is at least `gte`. propertyNames: $ref: '#/components/schemas/Name' additionalProperties: $ref: '#/components/schemas/ChoiceThreshold' ScoreJudgmentDefinition: type: object description: >- The answer has `score`, the probability-weighted mean of level values, and `dist`. required: - type - levels allOf: - $ref: '#/components/schemas/JudgmentDefinitionCommon' properties: type: const: score levels: type: array description: Ordered; level values must be distinct. minItems: 2 maxItems: 10 items: $ref: '#/components/schemas/ScoreLevel' thresholds: type: object description: Each threshold is true when `score` is at least the value. propertyNames: $ref: '#/components/schemas/Name' additionalProperties: type: number ShadowValues: type: object description: The mean and histogram of `p` (bool) or `score` (score). required: - type - mean - histogram properties: type: type: string enum: - bool - score mean: type: number histogram: type: array description: >- Ten equal-width bins, from 0 to 1 for `p` and from the lowest to the highest level for `score`, lowest first. minItems: 10 maxItems: 10 items: $ref: '#/components/schemas/HistogramBin' ShadowOptions: type: object description: >- The mean probability per option, so the per-option shift is `candidate.dist` minus `current.dist`. required: - type - dist properties: type: const: choice dist: $ref: '#/components/schemas/Distribution' MetricInterval: type: object required: - lower - upper properties: lower: $ref: '#/components/schemas/Probability' upper: $ref: '#/components/schemas/Probability' FeatureRef: type: string description: | Entities, E1 (§6.5.1). An aggregate the recipe declares, as `.count` or `.()`, such as `tickets.count` or `invoices.sum(state.amount)`. pattern: >- ^[A-Za-z0-9_-]+\.(count|(sum|min|max|latest)\((state|attributes)(\.[^.()]+)+\))$ AttributeFilter: type: object description: | Entities, E1 (§6.5). Which documents something applies to, by their attributes. Each key is an attribute path; a value is equality, and a list of values is `In`, as in query filters. Keys combine with `And`. minProperties: 1 maxProperties: 8 propertyNames: type: string pattern: ^attributes\.[A-Za-z0-9_-]+$ additionalProperties: oneOf: - $ref: '#/components/schemas/AttributeFilterValue' - type: array minItems: 1 items: $ref: '#/components/schemas/AttributeFilterValue' JudgmentDefinitionCommon: type: object required: - name - question properties: name: $ref: '#/components/schemas/Name' question: type: string minLength: 1 criteria: type: string context: $ref: '#/components/schemas/ContextRecipe' engine: $ref: '#/components/schemas/EngineRef' description: >- Must be `active` in the registry. May be omitted only when the namespace has a `default_engine`. horizon: $ref: '#/components/schemas/Horizon' description: | How far before `observed_at` an outcome's prediction was made, such as `30d` for "churned within 30 days". Part of the definition, so changing it creates a version. Defaults to `0s`, which joins labelled examples to current answers. applies_to: $ref: '#/components/schemas/AttributeFilter' description: | Entities, E1 (§6.5). The judgment judges, answers and bills only documents whose attributes match. A document that does not match has no answer for it: `answers` omits it. Part of the definition, so changing it creates a version. JudgmentPart: type: object description: One part of a composite judgment (§6.5.1). additionalProperties: false required: - name - question properties: name: $ref: '#/components/schemas/Name' question: type: string minLength: 1 description: A narrow yes/no question about the document. ChoiceOption: type: object additionalProperties: false required: - value properties: value: type: string minLength: 1 not: const: none_of_the_above description: type: string ScoreLevel: type: object additionalProperties: false required: - value - label properties: value: type: integer label: type: string minLength: 1 description: type: string HistogramBin: type: object description: Values in `[lower, upper)`; the last bin includes `upper`. required: - lower - upper - count properties: lower: type: number upper: type: number count: type: integer format: int64 minimum: 0 Distribution: type: object description: Probability per option or level value. Sums to 1. additionalProperties: $ref: '#/components/schemas/Probability' AttributeFilterValue: type: - string - number - boolean ContextRecipe: type: object description: | What the engine sees. Omitting it sends the whole `state`, subject to the engine limit; allowed but not recommended. additionalProperties: false properties: fields: type: array description: Paths included verbatim, in order. items: type: string pattern: ^(state|attributes)(\.[^.]+)+$ last_n: type: object description: Per array path, keep the last n elements. additionalProperties: type: integer minimum: 1 window: type: object description: >- Per array path whose elements have a timestamp field `at`, keep elements within the window. additionalProperties: $ref: '#/components/schemas/Duration' max_tokens: type: integer minimum: 1 description: Hard cap; the compiler truncates from the end of the last field. related: type: object description: | Entities, E1 (§6.5.2). Up to 4 relations, by name: documents in the same namespace that point at the judged one or, from E3a, the one document it points at. Each renders as a `related.` entry after `fields`, and a write that changes what it renders makes the judged document's answer `pending` until it is judged again (for a referenced document, only inside the re-judge scope). minProperties: 1 maxProperties: 4 propertyNames: $ref: '#/components/schemas/Name' additionalProperties: $ref: '#/components/schemas/Relation' EngineRef: type: object description: An exact engine version. No aliases. required: - name - version properties: name: type: string enum: - jev - laya version: type: string minLength: 1 Horizon: type: string description: | How long before an outcome is observed the prediction it measures was made: an outcome is joined to the evaluation current at `observed_at` minus the horizon (§6.10). A whole number and a unit (`s`, `m`, `h` or `d`); `0s` for labelled examples. pattern: ^(0|[1-9][0-9]*)[smhd]$ default: 0s Relation: type: object description: | Entities, E1 (§6.5.2). Documents that point at the judged document. They are ordered newest `created_at` first: `window` keeps those created within the window, then `last_n` keeps the newest n. At least one of `last_n` and `window` is required, and at least one of `fields` and `aggregate`. A relation reads at most the newest 1,000 documents. Entities, E3a: with `join: {theirs: "id", mine: "attributes."}` the relation reads the one document the judged document points at, its referenced document, and takes neither `last_n` nor `window`. additionalProperties: false required: - match - join allOf: - description: A relation renders fields or aggregates. if: not: required: - fields then: required: - aggregate - description: | A relation over the documents that point at the judged one is bounded by `last_n` or `window`; one that reads the referenced document takes neither. if: properties: join: type: object properties: mine: const: id then: anyOf: - required: - last_n - required: - window else: not: anyOf: - required: - last_n - required: - window properties: match: $ref: '#/components/schemas/AttributeFilter' description: Which documents the relation reads. join: $ref: '#/components/schemas/RelationJoin' last_n: type: integer minimum: 1 maximum: 1000 description: >- Keep the newest n. Not on a relation that reads a referenced document. window: $ref: '#/components/schemas/Duration' description: >- Keep documents created within this long before the later of the judged document's newest write and its newest related creation, at the answer's watermark. An edit to an old related document never moves the window. Not on a relation that reads a referenced document. fields: type: array description: >- Paths to include from each document, rendered under `records`, newest first. From entities E3a, an entry may be a `BandedField`, which renders a number as its band. minItems: 1 items: oneOf: - $ref: '#/components/schemas/RelationField' - $ref: '#/components/schemas/BandedField' aggregate: $ref: '#/components/schemas/Aggregate' RelationJoin: description: | How a related document is matched to the judged one: one side is `id`, the other an attribute holding an id. A join with an attribute on both sides, documents sharing a key, is E3b and `invalid_request`. oneOf: - $ref: '#/components/schemas/ReferringJoin' - $ref: '#/components/schemas/ReferencedJoin' RelationField: type: string description: A path in a related document, `id`, `created_at` or `updated_at`. pattern: ^((state|attributes)(\.[^.]+)+|id|created_at|updated_at)$ BandedField: type: object description: | Entities, E3a (§6.5.2). A number rendered as its band rather than its value, in any relation. A number renders as the label whose position is how many cut points are at or below it: with `bands: [0.3, 0.7]`, 0.29 is `labels[0]`, 0.3 is `labels[1]` and 0.7 or more `labels[2]`. A value that is not a number renders unchanged, and a missing one is left out. Only a move to another band changes the context, so a move inside one neither fans out nor costs an evaluation. `bands` must be strictly increasing and `labels` must have exactly one more entry; the server refuses anything else with `invalid_request`. additionalProperties: false required: - path - bands - labels properties: path: type: string description: A `state` or `attributes` path. pattern: ^(state|attributes)(\.[^.]+)+$ bands: type: array description: Cut points, strictly increasing. minItems: 1 maxItems: 9 items: type: number labels: type: array description: One label per band, lowest first, so one more than `bands`. minItems: 2 maxItems: 10 items: type: string minLength: 1 maxLength: 64 Aggregate: type: object description: | Numbers over the relation's documents, after `match`, `window` and `last_n`, rendered as keys of its `related.` entry, such as `count` and `sum(state.amount)`. At most 8 paths in total. `sum`, `min` and `max` skip values that are not numbers; `sum` of nothing is 0, and `min` and `max` of nothing are null. `latest` is the value in the newest document that has one, of any JSON type. additionalProperties: false minProperties: 1 properties: count: const: true description: How many documents the relation selected. sum: $ref: '#/components/schemas/AggregatePaths' min: $ref: '#/components/schemas/AggregatePaths' max: $ref: '#/components/schemas/AggregatePaths' latest: $ref: '#/components/schemas/AggregatePaths' ReferringJoin: type: object description: | Entities, E1 (§6.5.2). The documents that point at the judged one: a related document belongs to the judged document whose `id` equals its `theirs` attribute. additionalProperties: false required: - theirs - mine properties: theirs: type: string description: >- The related document's attribute that holds the judged document's id, such as `attributes.parent_id`. pattern: ^attributes\.[A-Za-z0-9_-]+$ mine: const: id ReferencedJoin: type: object description: | Entities, E3a (§6.5.2). The one document the judged document points at, its referenced document: the document whose `id` equals the judged document's `mine` attribute. A change to what the relation renders of it re-judges the judged documents that point at it and are inside `freshness.fanout.scope`. additionalProperties: false required: - theirs - mine properties: theirs: const: id mine: type: string description: >- The judged document's attribute that holds the referenced document's id, such as `attributes.group_id`. pattern: ^attributes\.[A-Za-z0-9_-]+$ AggregatePaths: type: array minItems: 1 maxItems: 8 uniqueItems: true items: type: string description: A `state` or `attributes` path. Keys in it cannot contain `(` or `)`. pattern: ^(state|attributes)(\.[^.()]+)+$ headers: RateLimitLimit: description: Requests allowed per second for this key. schema: type: integer minimum: 0 RateLimitRemaining: description: Requests left in the current window. schema: type: integer minimum: 0 RetryAfter: description: Seconds to wait before retrying. schema: type: integer minimum: 0 responses: InvalidRequest: description: '`invalid_request`: the request is malformed or breaks a rule of §6.' content: application/json: schema: $ref: '#/components/schemas/Error' Unauthorized: description: '`unauthorized`: the key is missing, unknown or revoked.' content: application/json: schema: $ref: '#/components/schemas/Error' Forbidden: description: '`forbidden`: the key''s role or namespace prefix does not allow this.' content: application/json: schema: $ref: '#/components/schemas/Error' NotFound: description: '`not_found`: the namespace, document, judgment or job does not exist.' content: application/json: schema: $ref: '#/components/schemas/Error' Conflict: description: >- `conflict`: the namespace is being deleted, an object a read needs was compacted away and the read (or the query, without its cursor) must restart (§7.5.11), or the judgment is inherited from a template and this namespace cannot create, activate or delete it (§7.7). content: application/json: schema: $ref: '#/components/schemas/Error' TooLarge: description: >- `too_large`: the request body (on any route), a document, or the estimated query scan is over its limit. `details` carries the estimate for queries. It also answers a write that would take `default/quickstart` past its size cap (10,000 documents or 100 MB), with `details.max_documents` and `details.max_bytes`. content: application/json: schema: $ref: '#/components/schemas/Error' EngineVersionUnavailable: description: >- `engine_version_unavailable`: the engine version is not active in the registry. content: application/json: schema: $ref: '#/components/schemas/Error' RateLimited: description: >- `rate_limited`: slow down. `Retry-After` says for how long, except when a daily allowance is used up: then `details.resets_at` says when it renews, and there is no `Retry-After`. headers: X-RateLimit-Limit: $ref: '#/components/headers/RateLimitLimit' X-RateLimit-Remaining: $ref: '#/components/headers/RateLimitRemaining' Retry-After: $ref: '#/components/headers/RetryAfter' content: application/json: schema: $ref: '#/components/schemas/Error' Internal: description: >- `internal`: something failed on our side. Nothing partial is returned, and a request never commits part of itself. Most `internal` errors committed nothing. A write can instead fail with a message saying it may have been committed, when we could not tell whether its batch landed (§7.2.10): it then committed all of it or none of it. Retrying a write is safe either way, because writes are idempotent. The message is generic and carries a request id, also in `details.request_id`, which support uses to find the cause. content: application/json: schema: $ref: '#/components/schemas/Error' securitySchemes: apiKey: type: http scheme: bearer description: | An organization API key. Keys carry a role (`read_write` or `read_only`) and may be restricted to a namespace prefix such as `acme/*`, or to one namespace such as `acme/prod/tenant_1`. A prefix matches on a `/` boundary: `acme/prod/tenant_1*` covers `acme/prod/tenant_1` and everything under `acme/prod/tenant_1/`, never `acme/prod/tenant_12`. ````