Skip to main content
An engine’s probabilities are its own. To know how good a judgment is on your data, give Vainona your answers to the same question. They are called outcomes. From them you get a calibration report, calibrated probabilities on every answer, and threshold recommendations. A shadow report shows what a new version of the question would change before you switch to it.

​
Post labelled examples

A labelled example is an outcome with the judgment’s default horizon of 0s. Write the documents, then post their labels. Each label is joined to the evaluation of the document revision that was current at observed_at, so a label posted before that evaluation finishes still joins it. Post them to POST /namespaces/{ns}/outcomes:
value is what the answer should have been:
  • A value that does not fit the judgment’s type is invalid_request. An unknown judgment is not_found.
  • An outcome for a document that has no evaluation of the judgment is kept, but counts nowhere.
  • Outcomes are append-only. Retries are safe: the same document, judgment, value and observed_at is stored as one outcome.
  • Outcomes belong to a namespace. Post them on the namespace’s own path, never on a template prefix.

​
Upload labels as a CSV

On the dashboard, open the judgment and go to its Labels tab. Upload a CSV with the header document_id,value and an optional observed_at column in RFC 3339:
Your browser checks every row against the judgment’s type and lists the row errors before anything is sent. Valid rows are posted in batches of 1,000. A row without observed_at uses the time of the upload.

​
Read the calibration report

GET /namespaces/{ns}/judgments/{name}/calibration returns the report for the active version. It is ns.judgments.calibration(name) in both SDKs, and the Calibration tab on the judgment’s page in the dashboard.
Epochs are listed newest first. Each one gives:
  • outcomes, the outcomes joined to this epoch’s evaluations.
  • method, how the epoch is calibrated, and fitted_at, when: Calibration is refitted every night.
  • raw and calibrated, the same metrics before and after calibration:
    • accuracy. A bool answer counts as true when p is at least 0.5. For choice and score, the most probable option or level must match the outcome exactly.
    • expected_calibration_error, how far the stated probabilities are from how often things came true. Lower is better.
    • log_loss. Lower is better.
    • reliability, 10 equal-width bins such as {"lower": 0.8, "upper": 0.9, "count": 137, "mean_predicted": 0.845, "observed": 0.883}. A well calibrated judgment has observed close to mean_predicted in every bin. For choice and score, each answer is binned by the probability of its most probable option or level. Empty bins have null means.
    • mean_level_distance, for score judgments only: how many levels the most probable level is from the observed one, on average.
The report is computed when you ask for it, so the raw metrics include outcomes you posted a minute ago. The fit itself changes when it is refitted overnight.

​
Epochs

An epoch is the engine_version recorded on an evaluation. An exact engine version is one epoch. Jev current starts a new epoch each time its behaviour changes, labelled like current+2026-09-24.1; see the engine page. Calibration is fitted per judgment version and per epoch, because a different model needs a different fit. After a drift, the new epoch starts with no outcomes. Until it has 100, answers under it use the previous epoch’s calibration and say so with "from_previous_epoch": true.

​
Calibrated answers

Once a judgment’s fit rests on 100 outcomes, every answer carries a calibrated object beside the raw numbers:
  • It has the answer type’s own fields: p for bool; value, dist and escape_p for choice; score and dist for score. It adds method, outcomes (what the fit rests on) and from_previous_epoch.
  • It never replaces p, dist or score, which stay the engine’s raw output.
  • It is absent below 100 outcomes, not null.
  • It is computed when the answer is read, from the current fit for the answer’s version and epoch. A refit changes it without recomputing anything.
  • Thresholds, filters and ranking use the raw fields.

​
Change a question safely with a shadow report

To change a judgment’s question, criteria, context or engine, create version n+1 by posting the definition again under the same name. It stays inactive. Then activate it with POST /namespaces/{ns}/judgments/needs_escalation/activate:
When version 4’s engine or definition differs from the active one, the response is 202 with a shadow job in awaiting_confirm. The job judges a random sample of 1,000 documents under version 4, or every document if there are fewer. While it samples, report is null and progress.documents_done counts the sampled documents. When it is done, the report compares the two versions on the same documents:
  • current is the active version’s answers, and candidate the new version’s results. For bool and score each side has a mean and a 10-bin histogram of p or score. For choice each side has dist, the mean probability of each option, so you can read the shift option by option.
  • threshold_flips counts, for each named threshold, the documents that would go from false to true and from true to false. The new side uses the thresholds that will apply once the version is active.
  • recompute estimates backfilling every document under the new version: documents, tokens, judgment units, cost and duration.
Then decide:
  • POST /jobs/{id}/confirm (db.jobs.confirm(id)) switches to the new version. You can confirm before the report is done. The job shows running, then done about a second later, once the switch is committed.
  • POST /jobs/{id}/cancel leaves the active version as it is.
  • {"version": 4, "force": true} on activate skips the report and switches at once. So does activate: true when you create a version. A composite judgment is the exception: its shadow job fits it on your labels, so it always runs.
Shadow evaluations are written to the evaluation log with shadow: true. They never produce answers, never count toward calibration, and are not billed. After the switch, existing answers keep their old judgment_version until their documents change. To recompute them all, run a backfill; recompute is its estimate. Thresholds you gave with the new version replace the current ones when it becomes active. In the dashboard, the judgment page’s Overview shows the report, with Confirm, Cancel and Force.

​
Pick thresholds with the recommender

The recommender finds the threshold that meets a precision or recall target on your outcomes:
target is precision:<x> or recall:<x>. For a choice judgment, add the option, such as &option=fraud; the threshold is then on the raw dist[fraud], and a choice without option is refused with invalid_request. score judgments get no recommendation: the call is refused with invalid_request.
The response is always 200, and status says which of three it is:
  • A precision target gets the lowest threshold that meets it, which keeps the most recall.
  • A recall target gets the highest threshold that meets it, which keeps the most precision.
  • The curve has 101 points, thresholds 0.00 to 1.00 in steps of 0.01, each with its precision and recall. The recommended threshold is one of them. Precision is null where no answer reaches the threshold.
  • The outcomes are the current epoch’s. When it has fewer than 100, the previous epoch’s are used and from_previous_epoch is true.
Nothing changes until you apply it. Thresholds are settings, changed with PATCH /namespaces/{ns}/judgments/{name}. The thresholds you send replace the whole set, so include the ones you want to keep:
For a choice, a threshold names its option: {"fraud": {"value": "fraud", "gte": 0.62}}. {} removes every threshold. The change applies at the next read to every answer, including answers already computed. Nothing is recomputed, no version is created, and the change is recorded in your audit log. In the dashboard, the judgment’s Thresholds tab applies a recommendation in one click, after you confirm. It keeps your other thresholds.

​
Real-world outcomes with a horizon

Labelled examples say what the answer should have been at the time. Some questions are predictions, and the outcome arrives later. For “will this customer churn within 30 days?”, give the judgment a horizon:
Post the outcome when it happens:
Each outcome is joined to the evaluation that was current at observed_at minus the horizon: here, the answer as it stood on 15 September. That is the prediction the outcome measures. horizon is part of the definition, a whole number and a unit (s, m, h or d). It defaults to 0s, and changing it creates a new version.