> ## Documentation Index > Fetch the complete documentation index at: https://docs.vainona.ai/llms.txt > Use this file to discover all available pages before exploring further. # Measure, improve, tune > Find out how good a judgment is on your data, change it safely, and pick thresholds from your own outcomes. export const productName = "Vainona"; An engine's probabilities are its own. To know how good a judgment is on your data, give {productName} your answers to the same question. They are called outcomes. From them you get a calibration report, calibrated probabilities on every answer, and threshold recommendations. A shadow report shows what a new version of the question would change before you switch to it. ## Post labelled examples A labelled example is an outcome with the judgment's default `horizon` of `0s`. Write the documents, then post their labels. Each label is joined to the evaluation of the document revision that was current at `observed_at`, so a label posted before that evaluation finishes still joins it. Post them to `POST /namespaces/{ns}/outcomes`: ```json theme={null} { "outcomes": [ {"document_id": "t_123", "judgment": "needs_escalation", "value": true, "observed_at": "2026-10-01T09:00:00Z"}, {"document_id": "t_124", "judgment": "needs_escalation", "value": false, "observed_at": "2026-10-01T09:00:00Z"} ] } ``` ```ts TypeScript theme={null} await ns.outcomes.append([ { document_id: "t_123", judgment: "needs_escalation", value: true, observed_at: "2026-10-01T09:00:00Z" }, { document_id: "t_124", judgment: "needs_escalation", value: false, observed_at: "2026-10-01T09:00:00Z" }, ]); ``` ```python Python theme={null} ns.outcomes.append([ {"document_id": "t_123", "judgment": "needs_escalation", "value": True, "observed_at": "2026-10-01T09:00:00Z"}, {"document_id": "t_124", "judgment": "needs_escalation", "value": False, "observed_at": "2026-10-01T09:00:00Z"}, ]) ``` `value` is what the answer should have been: | Judgment type | `value` | | ------------- | --------------------------------------------------------------------------- | | `bool` | `true` or `false` | | `choice` | An option's `value`, such as `"billing"`. `"none_of_the_above"` is allowed. | | `score` | A level's `value`, as a JSON integer, such as `4` | * A value that does not fit the judgment's type is `invalid_request`. An unknown judgment is `not_found`. * An outcome for a document that has no evaluation of the judgment is kept, but counts nowhere. * Outcomes are append-only. Retries are safe: the same document, judgment, value and `observed_at` is stored as one outcome. * Outcomes belong to a namespace. Post them on the namespace's own path, never on a [template](/guides/templates) prefix. ### Upload labels as a CSV On the dashboard, open the judgment and go to its **Labels** tab. Upload a CSV with the header `document_id,value` and an optional `observed_at` column in RFC 3339: ```text theme={null} document_id,value,observed_at t_123,true,2026-10-01T09:00:00Z t_124,false,2026-10-01T09:00:00Z t_125,true ``` Your browser checks every row against the judgment's type and lists the row errors before anything is sent. Valid rows are posted in batches of 1,000. A row without `observed_at` uses the time of the upload. ## Read the calibration report `GET /namespaces/{ns}/judgments/{name}/calibration` returns the report for the active version. It is `ns.judgments.calibration(name)` in both SDKs, and the **Calibration** tab on the judgment's page in the dashboard. ```json theme={null} { "judgment": "needs_escalation", "type": "bool", "version": 3, "outcomes": 1469, "epochs": [ { "engine": "jev", "engine_version": "current+2026-09-24.1", "outcomes": 1432, "method": "isotonic", "fitted_at": "2026-10-02T03:00:00Z", "raw": {"accuracy": 0.801, "expected_calibration_error": 0.159, "log_loss": 0.441, "reliability": [...]}, "calibrated": {"accuracy": 0.837, "expected_calibration_error": 0.008, "log_loss": 0.344, "reliability": [...]} } ] } ``` Epochs are listed newest first. Each one gives: * **`outcomes`**, the outcomes joined to this epoch's evaluations. * **`method`**, how the epoch is calibrated, and **`fitted_at`**, when: | Outcomes | `method` | | -------------- | ----------------------------------------------------------------------------------------------------------- | | Fewer than 100 | `null`. There is no calibration yet, and `calibrated` is `null`. | | 100 to 999 | `platt`: Platt scaling for `bool`, temperature scaling over the whole distribution for `choice` and `score` | | 1,000 or more | `isotonic`: isotonic regression | Calibration is refitted every night. * **`raw`** and **`calibrated`**, the same metrics before and after calibration: * **`accuracy`**. A `bool` answer counts as `true` when `p` is at least 0.5. For `choice` and `score`, the most probable option or level must match the outcome exactly. * **`expected_calibration_error`**, how far the stated probabilities are from how often things came true. Lower is better. * **`log_loss`**. Lower is better. * **`reliability`**, 10 equal-width bins such as `{"lower": 0.8, "upper": 0.9, "count": 137, "mean_predicted": 0.845, "observed": 0.883}`. A well calibrated judgment has `observed` close to `mean_predicted` in every bin. For `choice` and `score`, each answer is binned by the probability of its most probable option or level. Empty bins have `null` means. * **`mean_level_distance`**, for `score` judgments only: how many levels the most probable level is from the observed one, on average. The report is computed when you ask for it, so the raw metrics include outcomes you posted a minute ago. The fit itself changes when it is refitted overnight. ### Epochs An epoch is the `engine_version` recorded on an evaluation. An exact engine version is one epoch. Jev `current` starts a new epoch each time its behaviour changes, labelled like `current+2026-09-24.1`; see the [engine page](/engines/jev-current). Calibration is fitted per judgment version and per epoch, because a different model needs a different fit. After a drift, the new epoch starts with no outcomes. Until it has 100, answers under it use the previous epoch's calibration and say so with `"from_previous_epoch": true`. ### Calibrated answers Once a judgment's fit rests on 100 outcomes, every answer carries a `calibrated` object beside the raw numbers: ```json theme={null} { "needs_escalation": { "type": "bool", "p": 0.91, "calibrated": {"p": 0.84, "method": "isotonic", "outcomes": 1432, "from_previous_epoch": false}, "thresholds": {"escalate": true}, "freshness": "fresh", "engine_version": "current+2026-09-24.1" } } ``` * It has the answer type's own fields: `p` for `bool`; `value`, `dist` and `escape_p` for `choice`; `score` and `dist` for `score`. It adds `method`, `outcomes` (what the fit rests on) and `from_previous_epoch`. * It never replaces `p`, `dist` or `score`, which stay the engine's raw output. * It is absent below 100 outcomes, not `null`. * It is computed when the answer is read, from the current fit for the answer's version and epoch. A refit changes it without recomputing anything. * Thresholds, filters and ranking use the raw fields. ## Change a question safely with a shadow report To change a judgment's question, criteria, context or engine, create version n+1 by posting the definition again under the same name. It stays inactive. Then activate it with `POST /namespaces/{ns}/judgments/needs_escalation/activate`: ```json theme={null} {"version": 4} ``` When version 4's engine or definition differs from the active one, the response is `202` with a `shadow` job in `awaiting_confirm`. The job judges a random sample of 1,000 documents under version 4, or every document if there are fewer. While it samples, `report` is `null` and `progress.documents_done` counts the sampled documents. When it is done, the report compares the two versions on the same documents: ```json theme={null} { "id": "job_01J...", "type": "shadow", "status": "awaiting_confirm", "namespace": "acme/prod/tenant_123", "judgment": "needs_escalation", "version": 4, "progress": {"documents_done": 1000}, "report": { "documents": 1000, "current": {"type": "bool", "mean": 0.31, "histogram": [...]}, "candidate": {"type": "bool", "mean": 0.27, "histogram": [...]}, "threshold_flips": {"escalate": {"to_true": 6, "to_false": 31}}, "recompute": {"documents": 1204332, "tokens": 2408664000, "judgment_units": 2408664, "cost_usd": 240.87, "duration_s": 107529} } } ``` * **`current`** is the active version's answers, and **`candidate`** the new version's results. For `bool` and `score` each side has a `mean` and a 10-bin `histogram` of `p` or `score`. For `choice` each side has `dist`, the mean probability of each option, so you can read the shift option by option. * **`threshold_flips`** counts, for each named threshold, the documents that would go from false to true and from true to false. The new side uses the thresholds that will apply once the version is active. * **`recompute`** estimates backfilling every document under the new version: documents, tokens, judgment units, cost and duration. Then decide: * **`POST /jobs/{id}/confirm`** (`db.jobs.confirm(id)`) switches to the new version. You can confirm before the report is done. The job shows `running`, then `done` about a second later, once the switch is committed. * **`POST /jobs/{id}/cancel`** leaves the active version as it is. * **`{"version": 4, "force": true}`** on activate skips the report and switches at once. So does `activate: true` when you create a version. A [composite judgment](/guides/composite-judgments) is the exception: its shadow job fits it on your labels, so it always runs. Shadow evaluations are written to the evaluation log with `shadow: true`. They never produce answers, never count toward calibration, and are not billed. After the switch, existing answers keep their old `judgment_version` until their documents change. To recompute them all, run a [backfill](/concepts/judgments#backfill); `recompute` is its estimate. Thresholds you gave with the new version replace the current ones when it becomes active. In the dashboard, the judgment page's **Overview** shows the report, with **Confirm**, **Cancel** and **Force**. ## Pick thresholds with the recommender The recommender finds the threshold that meets a precision or recall target on your outcomes: ```text theme={null} GET /namespaces/{ns}/judgments/needs_escalation/thresholds/recommend?target=precision:0.9 ``` ```ts TypeScript theme={null} const rec = await ns.judgments.recommendThreshold("needs_escalation", { target: "precision:0.9" }); ``` ```python Python theme={null} rec = ns.judgments.recommend_threshold("needs_escalation", target="precision:0.9") ``` `target` is `precision:` or `recall:`. For a `choice` judgment, add the option, such as `&option=fraud`; the threshold is then on the raw `dist[fraud]`, and a `choice` without `option` is refused with `invalid_request`. `score` judgments get no recommendation: the call is refused with `invalid_request`. ```json theme={null} { "status": "recommended", "threshold": 0.79, "precision": 0.9, "recall": 0.407, "interval": {"lower": 0.849, "upper": 0.935}, "outcomes": 1432, "from_previous_epoch": false, "curve": [{"threshold": 0.0, "precision": 0.293, "recall": 1.0}, ...] } ``` The response is always `200`, and `status` says which of three it is: | `status` | Means | Carries | | ------------------- | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | | `recommended` | A threshold meets the target | `threshold`, `precision` and `recall` at it, `interval` (the 95% Wilson interval of the targeted metric), `outcomes`, and the `curve` | | `unreachable` | No threshold meets the target | The `curve` only, so you can see what is reachable | | `insufficient_data` | Fewer than 100 outcomes | `outcomes` | * **A precision target** gets the lowest threshold that meets it, which keeps the most recall. * **A recall target** gets the highest threshold that meets it, which keeps the most precision. * **The curve** has 101 points, thresholds 0.00 to 1.00 in steps of 0.01, each with its precision and recall. The recommended threshold is one of them. Precision is `null` where no answer reaches the threshold. * **The outcomes** are the current epoch's. When it has fewer than 100, the previous epoch's are used and `from_previous_epoch` is `true`. Nothing changes until you apply it. Thresholds are settings, changed with `PATCH /namespaces/{ns}/judgments/{name}`. The `thresholds` you send **replace the whole set**, so include the ones you want to keep: ```json theme={null} {"thresholds": {"escalate": 0.79, "review": 0.5}} ``` ```ts TypeScript theme={null} await ns.judgments.update("needs_escalation", { thresholds: { escalate: 0.79, review: 0.5 } }); ``` ```python Python theme={null} ns.judgments.update("needs_escalation", thresholds={"escalate": 0.79, "review": 0.5}) ``` For a `choice`, a threshold names its option: `{"fraud": {"value": "fraud", "gte": 0.62}}`. `{}` removes every threshold. The change applies at the next read to every answer, including answers already computed. Nothing is recomputed, no version is created, and the change is recorded in your audit log. In the dashboard, the judgment's **Thresholds** tab applies a recommendation in one click, after you confirm. It keeps your other thresholds. ## Real-world outcomes with a horizon Labelled examples say what the answer should have been at the time. Some questions are predictions, and the outcome arrives later. For "will this customer churn within 30 days?", give the judgment a `horizon`: ```json theme={null} { "name": "will_churn", "type": "bool", "question": "Will this customer cancel their subscription within the next 30 days?", "context": {"fields": ["state.plan_history", "attributes.plan"], "window": {"state.events": "30d"}, "max_tokens": 4000}, "engine": {"name": "jev", "version": "current"}, "horizon": "30d", "freshness": {"policy": "periodic", "interval": "1d"}, "thresholds": {"at_risk": 0.6} } ``` Post the outcome when it happens: ```json theme={null} {"outcomes": [{"document_id": "c_881", "judgment": "will_churn", "value": true, "observed_at": "2026-10-15T00:00:00Z"}]} ``` Each outcome is joined to the evaluation that was current at `observed_at` minus the horizon: here, the answer as it stood on 15 September. That is the prediction the outcome measures. `horizon` is part of the definition, a whole number and a unit (`s`, `m`, `h` or `d`). It defaults to `0s`, and changing it creates a new version.