> ## Documentation Index > Fetch the complete documentation index at: https://docs.vainona.ai/llms.txt > Use this file to discover all available pages before exploring further. # Propose options for what the escape option holds > Relations (§6.5.4). For a fixed-option `choice` only: a `discover` job samples up to 200 documents in `window` whose answer is `none_of_the_above` (or whose `escape_p` is at or above the judgment's `escape` threshold), cut to 100,000 tokens together, asks a general-purpose model for up to `count` new options, and runs the proposed version in shadow over the sample. Its `proposal` is the current options plus the proposed ones, with how many escape documents each would absorb and how many stay unlabelled. It never creates a version: edit the options, create the version, and activate it through its shadow report. - **Opt-in.** `forbidden` until an org admin enables `suggestions`, because the sample goes to an LLM provider, a disclosed subprocessor, as for `suggest_parts`. - **Limits.** Free, and counted against `suggest_parts`' daily limits (`rate_limited`). - A `bool`, a `score` or an `options.from` judgment is `invalid_request`: none of them has a fixed escape to propose from. ## OpenAPI ````yaml /api-reference/openapi.yaml post /namespaces/{ns}/judgments/{name}/discover openapi: 3.1.0 info: title: Vainona API version: '1' license: name: Proprietary identifier: LicenseRef-Proprietary description: | The `/v1` contract from main spec §6. It is locked: changes within `/v1` are additive only, and anything breaking is `/v2`. - **Base URL.** `https://api.vainona.ai/v1`. - **Evolution.** Response objects may gain fields and enums may gain values. Clients must ignore what they do not know. Request objects reject unknown fields. - **Path segments.** Namespace names (`{ns}`) and document ids (`{id}`) may contain `/`. Send it percent-encoded as `%2F`, so that `acme/prod/tenant_123` is `acme%2Fprod%2Ftenant_123`. - **Templates (v1.5).** The judgment routes also take a namespace prefix ending in `/*`, such as `acme%2Fprod%2F*`. A judgment created there is a template: every namespace under the prefix inherits it, including ones created later. The most specific matching prefix wins (§7.7). - **Thresholds are settings (v1.5).** Named thresholds belong to the judgment, not to a version. They are evaluated at read time against the raw fields, so a change applies at once to every answer, with no recompute and no new version (§6.5). - **Entity judgments (entities, E1).** A judgment can read the documents that point at the judged one, through `context.related`, and apply only to documents matching `applies_to` (§6.5.2). Its answers carry a `watermark`, and creating one that runs `on_change` returns a replay estimate of its monthly cost until you confirm (§6.9). - **Referenced documents (entities, E3a).** A relation can also read the one document the judged document points at, through `join: {theirs: "id", mine: "attributes."}`. A change to what the relation renders of that document re-judges the judged documents that point at it and are inside `freshness.fanout.scope`: a fan-out. - **Relations.** A relation can read plain judgments' answers (banded or cut, never summed), or be a blocking relation over the documents that share a key, which a `choice` can choose among with `options.from`. A recipe can read the document's groups (`context.group`) and the document as the judgment last saw it (`context.previous`). A judgment that can fan out confirms its rolling limits once, at create; past them, work is deferred, reads `stale` with an event, and catches up on its own. Nothing waits for a confirm at runtime (§6.5.2, §6.5.3, §6.9). The E3a settings this retires (`job_above`, `confirm_jobs`, `auto_daily_limit`, `fanout` jobs) are marked `deprecated` here until servers stop returning them; requests that set them are refused. - **Idempotency.** Every write operation is idempotent by construction: a retried append adds nothing while its values are still in the array, and a retried upsert or patch sets the same content again (§6.4). `Idempotency-Key` is accepted on the mutating routes that declare it and, while the node still caches the original response, returns it verbatim. Marking or unmarking a non-production prefix, and starting this month's recipe-tuning run, change nothing when repeated, so they take none. - **Errors.** Every error is the `Error` envelope; the HTTP status follows the code (see each response). - **Rate limits.** Every response carries `X-RateLimit-Limit` and `X-RateLimit-Remaining`; a `rate_limited` response adds `Retry-After`, except a daily allowance that is used up, which says when it renews in `details.resets_at` instead. - **Usage.** Every response that bills carries `usage`. Judging is billed per judgment, engine-neutral, by size class: one judgment answered counts by the tokens of its compiled context and its question together, 1 if standard (up to 2,000 tokens), 4 if large (up to 8,000) and 16 if extra-large (up to 32,000, the engine maximum) (§9). No response ever carries an engine's price. servers: - url: https://api.vainona.ai/v1 security: - apiKey: [] tags: - name: Namespaces description: Namespaces, their settings and cache warming (§6.2, §6.11). - name: Documents description: Writes, point reads and queries (§6.3 to §6.8). - name: Judgments description: >- Judgment definitions, versions, settings, activation, backfill and template detach (§6.5, §6.9, §7.7). - name: Outcomes and calibration description: | Post what actually happened to judged documents, and read how well answers match it: the calibration report, calibrated answers and threshold recommendations (§6.10). See [calibration](/concepts/calibration). - name: Groups description: | Relations (§6.5.3, §7.7). Groups keep aggregates per value of a key attribute, for judgments to read as the population a document belongs to and for reporting; a template's tenant summary publishes each tenant's groups, banded, as a document the platform can judge. - name: Jobs description: >- Backfill, shadow, periodic, deletion, reference index, evaluation export, and from relations resync, group build, discovery and simulation jobs (§6.2, §6.9, §7.10.5). - name: Engines description: The engine registry (§6.5, §7.4.5). - name: Organization description: | Settings of the whole organization: prefixes marked non-production, which the tenant fee and template calibration pools leave out (§9). - name: Subscriptions description: | Saved queries that send an event when a document starts or stops matching: the query's filter grammar (§6.8), on a namespace or a template prefix, with the fields each event carries. - name: Webhooks description: | Webhook endpoints, the events feed, the delivery log, redelivery and recovery. Every event type and its body is under `webhooks`. Events are delivered at least once, signed as Standard Webhooks, and kept for 30 days. Webhooks are on every plan, and deliveries are not billed; plans set how many endpoints an organization has (§9). paths: /namespaces/{ns}/judgments/{name}/discover: parameters: - $ref: '#/components/parameters/Namespace' - $ref: '#/components/parameters/JudgmentName' post: tags: - Judgments summary: Propose options for what the escape option holds description: | Relations (§6.5.4). For a fixed-option `choice` only: a `discover` job samples up to 200 documents in `window` whose answer is `none_of_the_above` (or whose `escape_p` is at or above the judgment's `escape` threshold), cut to 100,000 tokens together, asks a general-purpose model for up to `count` new options, and runs the proposed version in shadow over the sample. Its `proposal` is the current options plus the proposed ones, with how many escape documents each would absorb and how many stay unlabelled. It never creates a version: edit the options, create the version, and activate it through its shadow report. - **Opt-in.** `forbidden` until an org admin enables `suggestions`, because the sample goes to an LLM provider, a disclosed subprocessor, as for `suggest_parts`. - **Limits.** Free, and counted against `suggest_parts`' daily limits (`rate_limited`). - A `bool`, a `score` or an `options.from` judgment is `invalid_request`: none of them has a fixed escape to propose from. operationId: discoverOptions parameters: - $ref: '#/components/parameters/IdempotencyKey' requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/DiscoverRequest' responses: '202': description: The `discover` job. Its `proposal` fills in when it is done. headers: X-RateLimit-Limit: $ref: '#/components/headers/RateLimitLimit' X-RateLimit-Remaining: $ref: '#/components/headers/RateLimitRemaining' content: application/json: schema: $ref: '#/components/schemas/Job' '400': $ref: '#/components/responses/InvalidRequest' '401': $ref: '#/components/responses/Unauthorized' '403': $ref: '#/components/responses/Forbidden' '404': $ref: '#/components/responses/NotFound' '413': $ref: '#/components/responses/TooLarge' '429': $ref: '#/components/responses/RateLimited' '500': $ref: '#/components/responses/Internal' components: parameters: Namespace: name: ns in: path required: true description: The namespace name, with any `/` sent as `%2F`. schema: $ref: '#/components/schemas/NamespaceName' JudgmentName: name: name in: path required: true schema: $ref: '#/components/schemas/Name' IdempotencyKey: name: Idempotency-Key in: header description: >- Returns the original response verbatim while the node still caches it. Correctness never depends on it. schema: type: string minLength: 1 schemas: DiscoverRequest: type: object description: Relations (§6.5.4). additionalProperties: false properties: window: $ref: '#/components/schemas/Duration' description: How far back to sample escape documents from. Defaults to `7d`. count: type: integer minimum: 1 maximum: 10 default: 5 description: The most new options to propose. Job: type: object description: | A job row (§7.10.5). A `shadow` job comes from activating a differing version without `force` (§6.9). It starts in `awaiting_confirm` and runs its sample while it waits; `report` fills in when the sample is done. `confirm` activates `version` and the job ends `done`; `cancel` leaves the active version unchanged. On a template prefix, `namespace` is the prefix. A `reference_index` job (entities, E1) builds the index an entity judgment's relations need (§6.5.2); it starts `running`, and `attribute` names the attribute it indexes. An `evaluation_export` job (§9 Plans) starts `running`, counts evaluations written in `progress.documents_done`, never spends, and has `export`. From relations: a `resync` job re-renders the readers of a changed threshold (`resync`) and touches the parents whose rendering flipped; a `group_build` job computes a new group's or aggregate's first aggregates with no fan-out (`group`); a `discover` job fills in `proposal` (§6.5.4); a `simulation` job fills in `simulation` (§6.5.2). Each starts `running`; `discover` and `simulation` never spend. E3a's `fanout` jobs are retired: none is created, and a server may still return an old one until it stops. required: - id - type - status - namespace - spend_usd - progress - created_at - updated_at properties: id: type: string type: type: string enum: - backfill - shadow - periodic - namespace_delete - reference_index - fanout - evaluation_export - resync - group_build - discover - simulation status: type: string enum: - estimating - awaiting_confirm - running - paused - done - failed - cancelled namespace: $ref: '#/components/schemas/NamespaceOrTemplate' judgment: $ref: '#/components/schemas/Name' estimate: oneOf: - $ref: '#/components/schemas/Estimate' - type: 'null' spend_usd: type: number minimum: 0 description: Judging billed by this job so far, in US dollars. progress: type: object required: - documents_done properties: documents_done: type: integer format: int64 minimum: 0 description: | Documents done so far. A `backfill` counts the documents it has judged, and only those in its scope (the judgment's `applies_to` and the backfill's `filters`), so the count climbs to `estimate.documents`. It passes over the rest without counting them. estimated_completion_at: oneOf: - $ref: '#/components/schemas/Timestamp' - type: 'null' error: type: - string - 'null' attribute: type: - string - 'null' pattern: ^attributes\.[A-Za-z0-9_-]+$ description: >- Entities, E1 (§6.5.2). `reference_index` jobs only, as `attributes.`. The attribute whose reference index the job builds. Null for other jobs. referenced: description: Retired with E3a's `fanout` jobs; null. deprecated: true oneOf: - $ref: '#/components/schemas/ReferencedChange' - type: 'null' resync: $ref: '#/components/schemas/ResyncDetail' description: Relations. `resync` jobs only. group: $ref: '#/components/schemas/Name' description: >- Relations. `group_build` jobs only, the group whose aggregates it computes. proposal: description: Relations. `discover` jobs only; null until done. oneOf: - $ref: '#/components/schemas/DiscoveryProposal' - type: 'null' simulation: description: Relations. `simulation` jobs only; null until done. oneOf: - $ref: '#/components/schemas/SimulationReport' - type: 'null' version: $ref: '#/components/schemas/JudgmentVersionNumber' description: Shadow jobs only. The version being activated. report: description: | Shadow jobs only. Null while the sample runs; `progress` counts the sampled documents. A composite version's job has a `CompositeShadowReport` (§6.5.1), and when it has too few labels it fails with an `error` that starts with `insufficient_labels`. oneOf: - $ref: '#/components/schemas/ShadowReport' - $ref: '#/components/schemas/CompositeShadowReport' - type: 'null' export: $ref: '#/components/schemas/EvaluationExport' description: >- `evaluation_export` jobs only (§9 Plans): the range, the evaluations written, and once done the download links. created_at: $ref: '#/components/schemas/Timestamp' updated_at: $ref: '#/components/schemas/Timestamp' NamespaceName: type: string description: >- Up to 256 bytes. `/` separates levels of the hierarchy, as in `acme/prod/tenant_123`. pattern: ^[A-Za-z0-9._:/-]+$ minLength: 1 maxLength: 256 Name: type: string description: >- A judgment, attribute or threshold name. Names are path segments in field references, so they never contain `.`. pattern: ^[A-Za-z0-9_-]+$ minLength: 1 maxLength: 128 Duration: type: string description: A whole number and a unit (`s`, `m`, `h` or `d`), such as `7d` or `1h`. pattern: ^[1-9][0-9]*[smhd]$ NamespaceOrTemplate: type: string description: | A namespace name, or a template prefix: a namespace path ending in `/*`, such as `acme/prod/*`, which every namespace under `acme/prod/` inherits judgments from (§7.7). Up to 256 bytes. pattern: ^[A-Za-z0-9._:/-]+(/\*)?$ minLength: 1 maxLength: 256 Estimate: type: object required: - documents - tokens - judgment_units - cost_usd - duration_s properties: documents: type: integer format: int64 minimum: 0 tokens: type: integer format: int64 minimum: 0 judgment_units: type: integer format: int64 minimum: 0 description: >- Estimated judgments (§9), each counted by its size class, from a sample of up to 1,000 documents. cost_usd: type: number minimum: 0 description: >- What the backfill is expected to cost you, the judgments at the graduated prices of the tiers your organization will be in, counting what it has already used this month (§9). duration_s: type: integer format: int64 minimum: 0 description: >- Expected wall-clock seconds if nothing else is judged meanwhile: the documents at the bulk rate the engine's rate limit allows now (a fan-out at most its share of the namespace's requests in flight), plus about 2 s for each page of 256 documents, which is most of a small backfill's time. Judging that comes first (new writes, catch-ups, other organizations' backfills, which take turns with it) makes it longer, so read it as the least to expect. downstream: type: array description: >- Relations (§6.9). A backfill of a judgment whose answers other judgments read also re-judges them where a rendered band or cut flips; one line per reader. Absent when it has none. items: $ref: '#/components/schemas/DownstreamLine' Timestamp: type: string format: date-time description: RFC 3339, UTC. ReferencedChange: type: object description: | Entities, E3a (§6.9). The change a `fanout` job fans out: a write to the referenced document that changed what the relation renders. While the job waits for confirm, a later change to the same document moves `revision` to it, and one confirm covers them all. required: - relation - document_id - revision properties: relation: $ref: '#/components/schemas/Name' document_id: $ref: '#/components/schemas/DocumentId' revision: $ref: '#/components/schemas/Revision' description: >- The referenced document's revision at the change. Every judged document the job re-judges gets an answer with a `watermark` at or above it. ResyncDetail: type: object description: Relations (§6.9). What a `resync` job re-renders. required: - thresholds_before - thresholds_after - readers properties: thresholds_before: $ref: '#/components/schemas/ThresholdSettings' thresholds_after: $ref: '#/components/schemas/ThresholdSettings' readers: type: array items: $ref: '#/components/schemas/Name' DiscoveryProposal: type: object description: | Relations (§6.5.4). A new version's `options`: the current ones plus the proposed ones, with a shadow run of that version over the sample. Nothing was created. required: - options - proposed - sampled - escape_documents - unlabelled properties: options: type: array description: Every option of the proposed version, current ones first. items: $ref: '#/components/schemas/ChoiceOption' proposed: type: array items: $ref: '#/components/schemas/ProposedOption' sampled: type: integer minimum: 0 maximum: 200 description: Escape documents in the sample. escape_documents: type: integer format: int64 minimum: 0 description: Escape documents in the window, sampled or not. unlabelled: type: integer minimum: 0 description: >- Sampled documents the proposed version still answered `none_of_the_above`. SimulationReport: type: object description: | Relations (§6.5.2). What the proposed referenced document would flip, estimated from a weighted sample, and never written. required: - population - sampled - thresholds - examples - excluded - fanout - larger_sample properties: population: type: integer format: int64 minimum: 0 description: In-scope judged documents that point at the document. sampled: type: integer minimum: 0 thresholds: type: object description: Per named threshold. additionalProperties: $ref: '#/components/schemas/SimulatedFlips' examples: type: array description: Sampled documents that flipped, up to 20. maxItems: 20 items: $ref: '#/components/schemas/DocumentId' excluded: type: object description: Sampled documents left out, by reason. properties: failed: type: integer minimum: 0 description: Either evaluation failed. out_of_scope: type: integer minimum: 0 description: Outside `freshness.fanout.scope` at the snapshot. fanout: $ref: '#/components/schemas/Estimate' description: >- The real fan-out if the document were written so, priced like a backfill of the in-scope documents. larger_sample: type: - integer - 'null' description: >- A sample size that would narrow the intervals enough to decide, when this one cannot; null otherwise. JudgmentVersionNumber: type: integer format: int32 minimum: 1 ShadowReport: type: object description: | What activating `version` would change, from a shadow sample of 1,000 random documents, or every document when there are fewer (§7.3.11). Shadow evaluations are in the evaluation log with `shadow: true` and never produce answers. `current` is the answers of the active version for the sampled documents, and `candidate` the shadow evaluations. required: - documents - current - candidate - threshold_flips - recompute properties: downstream: type: array description: >- Relations (§6.9). What activating the version costs the judgments that read its answers, from the sample's band and cut flips; one line per reader. Absent when it has none. items: $ref: '#/components/schemas/DownstreamLine' documents: type: integer format: int64 minimum: 0 description: Documents in the sample. current: $ref: '#/components/schemas/ShadowSide' candidate: $ref: '#/components/schemas/ShadowSide' threshold_flips: type: object description: | Per named threshold, the sampled documents whose threshold boolean would change. The candidate side uses the thresholds that apply once the version is active. propertyNames: $ref: '#/components/schemas/Name' additionalProperties: $ref: '#/components/schemas/ThresholdFlips' recompute: $ref: '#/components/schemas/Estimate' description: >- What a backfill of every document under the new version would cost, as a backfill estimate. CompositeShadowReport: type: object description: | The shadow report of a composite version (§6.5.1). The job judged up to 1,000 documents with outcomes for this judgment and fitted the combiner with 5-fold cross-validation, so every document is scored by a fit that never saw its label. The baseline gets the same treatment: its cut-off is fitted by the same folds on the same outcomes, never the raw 0.5, so splitting gets no credit for what a fitted threshold alone would give. `verdict` is `better` only when the composite's accuracy beats the baseline's by more than the interval of the difference. Otherwise it is `not_better`, and the threshold recommender on the baseline is the cheaper fix. `confirm` activates the version either way, with the combiner fitted on all the documents. required: - type - documents - labels - composite - baseline - difference - parts - verdict - recompute properties: type: const: composite documents: type: integer format: int64 minimum: 0 description: Labelled documents judged and scored. labels: type: object description: The documents by their outcome. required: - 'true' - 'false' properties: 'true': type: integer format: int64 minimum: 0 'false': type: integer format: int64 minimum: 0 composite: $ref: '#/components/schemas/CrossValidated' baseline: $ref: '#/components/schemas/Baseline' difference: type: object description: >- The composite's cross-validated accuracy minus the baseline's, with its 95% interval over the documents (paired). required: - accuracy - interval properties: accuracy: type: number minimum: -1 maximum: 1 interval: $ref: '#/components/schemas/DifferenceInterval' parts: type: array description: >- Each part's weight in the combiner fitted on all the documents, on standardised log-odds, so sizes compare. A negative weight means a yes points to `false`. items: $ref: '#/components/schemas/PartWeight' features: type: array description: >- Entities, E1 (§6.5.1). Each feature's weight in the same fit, on the same standardised scale as the parts'. items: $ref: '#/components/schemas/FeatureWeight' verdict: type: string enum: - better - not_better recompute: $ref: '#/components/schemas/Estimate' description: >- What a backfill of every document under the new version would cost, as a backfill estimate. Each part is billed as a judgment. EvaluationExport: type: object description: | An `evaluation_export` job's range and files (§9 Plans). `files` is empty until the job is `done`; then `GET /jobs/{id}` links each file until `expires_at`, when the first file is deleted. Other routes that return the job leave `files` empty. When links can't be made right now, the files are listed with `url` null and `links_unavailable` says so: read the job again later. required: - since - until - judgments - evaluations - files - expires_at - links_unavailable properties: since: $ref: '#/components/schemas/Timestamp' until: $ref: '#/components/schemas/Timestamp' judgments: description: The judgments exported; null for every judgment. oneOf: - type: array items: $ref: '#/components/schemas/Name' - type: 'null' evaluations: type: integer format: int64 minimum: 0 description: Evaluations written so far, the same as `progress.documents_done`. files: type: array items: $ref: '#/components/schemas/EvaluationExportFile' expires_at: description: >- When the files are deleted, 7 days after the first was written. Null until the job is done, and when it wrote no file. oneOf: - $ref: '#/components/schemas/Timestamp' - type: 'null' links_unavailable: type: - string - 'null' description: >- Why some or all of `files` have no `url` right now, with a request id for support. Null when every file has its link. The job and its files are unaffected; read the job again later for new links. Error: type: object required: - error properties: error: type: object required: - code - message properties: code: $ref: '#/components/schemas/ErrorCode' message: type: string details: type: object description: >- Code-specific detail, such as the scan estimate on a refused query. Every `internal` error has `request_id`. properties: suggested_context: description: >- On a create without a context recipe, the recipe suggested from the namespace's documents, or null when there are none (D-v15-612). oneOf: - $ref: '#/components/schemas/SuggestedContext' - type: 'null' DownstreamLine: type: object description: | Relations (§6.9). One reader's cost of a change to a judgment whose answers its relations read: the re-judgments the change would cause a month (for a one-off such as a backfill, in the month it runs), counted from the answer history's band and cut flips over the last 30 days, priced by the reader's size class at your tiers. required: - judgment - judgments_per_month - cost_usd properties: judgment: $ref: '#/components/schemas/Name' judgments_per_month: type: integer format: int64 minimum: 0 cost_usd: type: number minimum: 0 DocumentId: type: string description: Up to 128 bytes. pattern: ^[A-Za-z0-9._:/-]+$ minLength: 1 maxLength: 128 Revision: type: integer format: int64 minimum: 0 ThresholdSettings: type: object description: | The judgment's named thresholds, a setting rather than part of a version (§6.5). Each is a number for a bool judgment (true when `p` is at least it) or a score judgment (true when `score` is at least it), and `{value, gte}` for a choice judgment (true when `dist[value]` is at least `gte`). They are evaluated at read time against the raw fields, never the `calibrated` ones. A threshold that does not fit the judgment's type is `invalid_request`. propertyNames: $ref: '#/components/schemas/Name' additionalProperties: oneOf: - type: number - $ref: '#/components/schemas/ChoiceThreshold' ChoiceOption: type: object additionalProperties: false required: - value properties: value: type: string minLength: 1 not: const: none_of_the_above description: type: string ProposedOption: type: object required: - value - description - absorbs - sample_ids properties: value: type: string description: type: string absorbs: type: integer minimum: 0 description: >- Sampled escape documents the proposed version answered with this option. sample_ids: type: array description: The sampled documents the model said this option covers. items: $ref: '#/components/schemas/DocumentId' SimulatedFlips: type: object required: - sampled_on - sampled_off - estimated - interval properties: sampled_on: type: integer minimum: 0 description: Sampled documents the threshold would newly hold for. sampled_off: type: integer minimum: 0 description: Sampled documents it would stop holding for. estimated: type: integer format: int64 minimum: 0 description: Estimated flips in the population, either way. interval: $ref: '#/components/schemas/LiftInterval' ShadowSide: type: object description: One side of a shadow report, discriminated on the judgment `type`. oneOf: - $ref: '#/components/schemas/ShadowValues' - $ref: '#/components/schemas/ShadowOptions' discriminator: propertyName: type mapping: bool: $ref: '#/components/schemas/ShadowValues' score: $ref: '#/components/schemas/ShadowValues' choice: $ref: '#/components/schemas/ShadowOptions' ThresholdFlips: type: object required: - to_true - to_false properties: to_true: type: integer format: int64 minimum: 0 description: Documents false under the active version and true under the new one. to_false: type: integer format: int64 minimum: 0 description: Documents true under the active version and false under the new one. CrossValidated: type: object description: Out-of-fold scores over the report's documents. required: - accuracy - accuracy_interval - auc properties: accuracy: $ref: '#/components/schemas/Probability' description: >- Share of documents whose out-of-fold probability is on the right side of 0.5. accuracy_interval: $ref: '#/components/schemas/MetricInterval' auc: $ref: '#/components/schemas/Probability' description: ROC AUC of the out-of-fold probabilities. Baseline: type: object description: >- The fair baseline, scored like the composite. The active version (`version`), or for a first version the composite's `question` asked on its own (`version` null). required: - version - threshold - accuracy - accuracy_interval - auc allOf: - $ref: '#/components/schemas/CrossValidated' properties: version: oneOf: - $ref: '#/components/schemas/JudgmentVersionNumber' - type: 'null' threshold: description: >- The baseline's fitted cut-off on its `p`, fitted on all the documents. Null when a higher `p` does not mean `true` more often. oneOf: - $ref: '#/components/schemas/Probability' - type: 'null' DifferenceInterval: type: object required: - lower - upper properties: lower: type: number minimum: -1 maximum: 1 upper: type: number minimum: -1 maximum: 1 PartWeight: type: object required: - name - weight properties: name: $ref: '#/components/schemas/Name' weight: type: number FeatureWeight: type: object required: - name - weight properties: name: $ref: '#/components/schemas/FeatureRef' weight: type: number EvaluationExportFile: type: object description: One gzipped JSON Lines file of an evaluation export. required: - url - url_expires_at - evaluations - bytes properties: url: description: >- A signed download link. It needs no API key and works until `url_expires_at`. Null when the link can't be made right now (`links_unavailable` says why). oneOf: - type: string format: uri - type: 'null' url_expires_at: description: When `url` stops working; null with it. oneOf: - $ref: '#/components/schemas/Timestamp' - type: 'null' evaluations: type: integer format: int64 minimum: 1 description: Lines in the file. bytes: type: integer format: int64 minimum: 1 description: The file's size, compressed. ErrorCode: type: string description: | HTTP status by code: `invalid_request` 400, `unauthorized` 401, `budget_exceeded` 402, `plan_required` 402, `forbidden` 403, `not_found` 404, `conflict` 409, `too_large` 413, `engine_version_unavailable` 422, `insufficient_labels` 422, `rate_limited` 429, `internal` 500, `engine_unavailable` 503, `unavailable` 503. enum: - invalid_request - unauthorized - forbidden - not_found - conflict - too_large - rate_limited - budget_exceeded - plan_required - engine_unavailable - engine_version_unavailable - unavailable - insufficient_labels - internal SuggestedContext: type: object description: | The recipe suggested for a create without `context` (D-v15-612), in the refusal's `details.suggested_context`. From up to 100 of the documents the definition applies to: short text fields, numbers and small attributes are kept; ids, timestamps, URLs, base64 and long values are left out; arrays longer than 5 keep their last 5; and `max_tokens` caps the largest context at a multiple of 500, at most 2,000. The same documents always give the same recipe. required: - recipe - cost_per_answer - whole_state_cost_per_answer - sampled_documents - excluded properties: recipe: $ref: '#/components/schemas/ContextRecipe' cost_per_answer: type: - number - 'null' minimum: 0 description: >- Mean judgments per answer, counted by size class, over the sample with this recipe and the definition's question. The suggested `max_tokens` is at most 2,000, the standard ceiling, which keeps the context standard; the question counts too, so a long one can make an answer large. whole_state_cost_per_answer: type: - number - 'null' minimum: 0 description: The same with the whole `state`. sampled_documents: type: integer minimum: 1 excluded: type: array description: >- Each path left out, and why, so you can put back what the question needs. items: type: object required: - path - reason properties: path: type: string reason: type: string enum: - id - timestamp - url - binary - long - rare - too_many ChoiceThreshold: type: object additionalProperties: false required: - value - gte properties: value: type: string minLength: 1 gte: $ref: '#/components/schemas/Probability' LiftInterval: type: object required: - lower - upper properties: lower: type: number upper: type: number ShadowValues: type: object description: The mean and histogram of `p` (bool) or `score` (score). required: - type - mean - histogram properties: type: type: string enum: - bool - score mean: type: number histogram: type: array description: >- Ten equal-width bins, from 0 to 1 for `p` and from the lowest to the highest level for `score`, lowest first. minItems: 10 maxItems: 10 items: $ref: '#/components/schemas/HistogramBin' ShadowOptions: type: object description: >- The mean probability per option, so the per-option shift is `candidate.dist` minus `current.dist`. required: - type - dist properties: type: const: choice dist: $ref: '#/components/schemas/Distribution' Probability: type: number minimum: 0 maximum: 1 description: | Answers store probabilities as 32-bit floats and return them as the shortest decimal that stands for the stored value, so an engine's 0.01 reads 0.01. Thresholds, query filters and threshold recommendations compare that decimal (§7.1.6). MetricInterval: type: object required: - lower - upper properties: lower: $ref: '#/components/schemas/Probability' upper: $ref: '#/components/schemas/Probability' FeatureRef: type: string description: | Entities, E1 (§6.5.1). An aggregate the recipe declares, as `.count` or `.()`, such as `tickets.count` or `invoices.sum(state.amount)`. pattern: >- ^[A-Za-z0-9_-]+\.(count|(sum|min|max|latest)\((state|attributes)(\.[^.()]+)+\))$ ContextRecipe: type: object description: | What the engine sees. A create without it is refused with a suggested recipe, unless `use_context` asks for the suggestion or the whole `state` (D-v15-612); a version with no recipe sends the whole `state`, subject to the engine limit. additionalProperties: false properties: fields: type: array description: Paths included verbatim, in order. items: type: string pattern: ^(state|attributes)(\.[^.]+)+$ last_n: type: object description: Per array path, keep the last n elements. additionalProperties: type: integer minimum: 1 window: type: object description: >- Per array path whose elements have a timestamp field `at`, keep elements within the window. additionalProperties: $ref: '#/components/schemas/Duration' max_tokens: type: integer minimum: 1 description: Hard cap; the compiler truncates from the end of the last field. related: type: object description: | Entities, E1 (§6.5.2). Up to 4 relations, by name: documents in the same namespace that point at the judged one or, from E3a, the one document it points at. Each renders as a `related.` entry after `fields`, and a write that changes what it renders makes the judged document's answer `pending` until it is judged again (for a referenced document, only inside the re-judge scope; outside it, the answer is `stale` with `stale_reason` `referenced_changed`). minProperties: 1 maxProperties: 4 propertyNames: $ref: '#/components/schemas/Name' additionalProperties: $ref: '#/components/schemas/Relation' group: type: object description: | Relations (§6.5.3). Up to 4 groups of the namespace, by name: the aggregates of the judged document's key value, banded, and the document's own position in them. Each renders as a `group.` entry after `related`. A crossing of a band re-judges the key value's members, within the judgment's `freshness.fanout.crossings_per_month`. minProperties: 1 maxProperties: 4 propertyNames: $ref: '#/components/schemas/Name' additionalProperties: $ref: '#/components/schemas/GroupRead' previous: $ref: '#/components/schemas/PreviousRecipe' HistogramBin: type: object description: Values in `[lower, upper)`; the last bin includes `upper`. required: - lower - upper - count properties: lower: type: number upper: type: number count: type: integer format: int64 minimum: 0 Distribution: type: object description: Probability per option or level value. Sums to 1. additionalProperties: $ref: '#/components/schemas/Probability' Relation: type: object description: | Entities, E1 (§6.5.2). Documents that point at the judged document. They are ordered newest `created_at` first: `window` keeps those created within the window, then `last_n` keeps the newest n. At least one of `last_n` and `window` is required, and at least one of `fields` and `aggregate`. A relation reads at most the newest 1,000 documents. Entities, E3a: with `join: {theirs: "id", mine: "attributes."}` the relation reads the one document the judged document points at, its referenced document, and takes neither `last_n` nor `window`. Relations: with `join: {theirs: "attributes.", mine: "attributes."}` it is a blocking relation over the documents that share the judged document's key value. It needs `window`, takes `last_n` up to 254 (50 when a `choice` chooses among it with `options.from`), and a change to its block re-judges the block's judged documents. `fields`, `aggregate` and `match` may read plain judgments' answers (`answers..`), banded or cut. additionalProperties: false required: - match - join allOf: - description: A relation renders fields or aggregates. if: not: required: - fields then: required: - aggregate - description: >- A relation over the documents that point at the judged one is bounded by `last_n` or `window`. if: properties: join: type: object properties: mine: const: id then: anyOf: - required: - last_n - required: - window - description: >- A relation that reads the referenced document takes neither `last_n` nor `window`. if: properties: join: type: object properties: theirs: const: id then: not: anyOf: - required: - last_n - required: - window - description: A blocking relation needs `window`, and reads at most 254 documents. if: properties: join: $ref: '#/components/schemas/BlockingJoin' then: required: - window properties: last_n: type: integer maximum: 254 properties: match: $ref: '#/components/schemas/RelationMatch' description: Which documents the relation reads. join: $ref: '#/components/schemas/RelationJoin' last_n: type: integer minimum: 1 maximum: 1000 description: >- Keep the newest n. Not on a relation that reads a referenced document; at most 254 on a blocking relation, and 50 when `options.from` chooses among it. window: $ref: '#/components/schemas/Duration' description: >- Keep documents created within this long before the later of the judged document's newest write and its newest related creation, at the answer's watermark. An edit to an old related document never moves the window. Not on a relation that reads a referenced document. fields: type: array description: >- Paths to include from each document, rendered under `records`, newest first. From entities E3a, an entry may be a `BandedField`, which renders a number as its band. From relations, a plain judgment's `answers..value` and `answers..thresholds.` as they are, and its `.p`, `.score` and `.dist.