> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vainona.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Calibration

> How answers learn from what actually happened, so a probability means what it says.

export const productName = "Vainona";

An engine's probability is its own confidence, and confidence is not the same as being right. An engine that says 0.9 might be right only 80% of the time on your documents, or 97%. Calibration measures that on your own outcomes and corrects it, so a calibrated 0.9 comes true about 90% of the time.

It matters whenever you act on a probability: a threshold that sends a document to a person, an automation that runs above 0.95, a queue sorted by risk. With calibrated numbers, you can say how often those actions will be wrong.

## Outcomes are the input

An **outcome** is what actually happened to a judged document, posted with `POST /namespaces/{ns}/outcomes`. There are two kinds:

* **Labelled examples.** A person's answer to the question for a document, such as "this ticket did need escalation". Write the documents, post the labels, and each is joined to the document's current answer. This measures a judgment before you rely on it.
* **Real-world results.** What happened later, such as "this account churned". A judgment with a `horizon`, such as `30d`, joins each result to the answer that was current that long before it, so you calibrate against what the judgment predicted at the time.

Outcomes are append-only. Each counts for the judgment version and the engine epoch of the evaluation it joins.

## How the fit works

Calibration is fitted separately for each judgment version and each engine epoch, because a different question or a different model needs a different correction.

| Outcomes for that version and epoch | What happens                                                                                                                                                                     |
| ----------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Fewer than 100                      | No calibration. Answers carry only the engine's raw numbers.                                                                                                                     |
| 100 to 999                          | Platt scaling for `bool`, and temperature scaling over the whole distribution for `choice` and `score`: a smooth correction with few parameters, which is steady on little data. |
| 1,000 or more                       | Isotonic regression: a flexible correction that follows the engine's actual pattern of over- and under-confidence.                                                               |

The first fit runs shortly after a judgment's first outcomes arrive, and it is refitted every night after that, so new outcomes improve it over time.

## What changes in an answer

Once a fit rests on 100 outcomes, each answer carries a `calibrated` object beside the raw numbers:

```json theme={null}
{
  "needs_escalation": {
    "type": "bool",
    "p": 0.91,
    "calibrated": {"p": 0.84, "method": "isotonic", "outcomes": 1432, "from_previous_epoch": false},
    "freshness": "fresh"
  }
}
```

* The raw `p`, `dist` and `score` never change. `calibrated` sits beside them.
* It is computed when the answer is read, so a nightly refit updates every answer without re-judging anything.
* Thresholds, filters and ranking use the raw numbers, so a refit never moves a document across a threshold on its own. To act on calibrated numbers, pick thresholds with the [recommender](/guides/measure-improve-tune#pick-thresholds-with-the-recommender), which works from your outcomes.

## What calibration can and cannot do

* **It makes probabilities honest.** After calibration, the stated probability matches how often answers come true, measured on your data. The [calibration report](/guides/measure-improve-tune#read-the-calibration-report) shows both, before and after.
* **It does not change the order of documents.** The correction only ever maps a higher raw probability to an equal or higher calibrated one. So it cannot make the engine better at telling likely documents from unlikely ones. If the right documents are not near the top of the raw ranking, a better question or [context recipe](/guides/context-recipes) is what helps.
* **It can change a yes or no at 0.5.** Accuracy before and after can differ, and the report shows both.

## When the engine changes

Jev `current` follows its provider's model, and each change in its behaviour that we detect starts a new **epoch**, recorded on every answer's `engine_version`. A new epoch starts with no outcomes. Until it has 100, its answers use the previous epoch's calibration and say so with `"from_previous_epoch": true`. Keep posting outcomes and the new epoch gets its own fit. An exact engine version never changes, so it is one epoch for its whole life.

A new judgment version starts fresh too. It needs its own 100 outcomes before its answers carry `calibrated`.

## Templates and composites

* A [template](/guides/templates) judgment's report on its prefix pools the outcomes of every namespace under it, and inherited answers use that pooled fit.
* A [composite judgment](/guides/composite-judgments) has no `calibrated` object. Its `p` already comes from weights fitted on your labels, refitted nightly in the same run.

To post outcomes, read the report and pick thresholds, see [measure, improve and tune](/guides/measure-improve-tune).