Skip to main content
An engine’s probability is its own confidence, and confidence is not the same as being right. An engine that says 0.9 might be right only 80% of the time on your documents, or 97%. Calibration measures that on your own outcomes and corrects it, so a calibrated 0.9 comes true about 90% of the time. It matters whenever you act on a probability: a threshold that sends a document to a person, an automation that runs above 0.95, a queue sorted by risk. With calibrated numbers, you can say how often those actions will be wrong.

​
Outcomes are the input

An outcome is what actually happened to a judged document, posted with POST /namespaces/{ns}/outcomes. There are two kinds:
  • Labelled examples. A person’s answer to the question for a document, such as “this ticket did need escalation”. Write the documents, post the labels, and each is joined to the document’s current answer. This measures a judgment before you rely on it.
  • Real-world results. What happened later, such as “this account churned”. A judgment with a horizon, such as 30d, joins each result to the answer that was current that long before it, so you calibrate against what the judgment predicted at the time.
Outcomes are append-only. Each counts for the judgment version and the engine epoch of the evaluation it joins.

​
How the fit works

Calibration is fitted separately for each judgment version and each engine epoch, because a different question or a different model needs a different correction. The first fit runs shortly after a judgment’s first outcomes arrive, and it is refitted every night after that, so new outcomes improve it over time.

​
What changes in an answer

Once a fit rests on 100 outcomes, each answer carries a calibrated object beside the raw numbers:
  • The raw p, dist and score never change. calibrated sits beside them.
  • It is computed when the answer is read, so a nightly refit updates every answer without re-judging anything.
  • Thresholds, filters and ranking use the raw numbers, so a refit never moves a document across a threshold on its own. To act on calibrated numbers, pick thresholds with the recommender, which works from your outcomes.

​
What calibration can and cannot do

  • It makes probabilities honest. After calibration, the stated probability matches how often answers come true, measured on your data. The calibration report shows both, before and after.
  • It does not change the order of documents. The correction only ever maps a higher raw probability to an equal or higher calibrated one. So it cannot make the engine better at telling likely documents from unlikely ones. If the right documents are not near the top of the raw ranking, a better question or context recipe is what helps.
  • It can change a yes or no at 0.5. Accuracy before and after can differ, and the report shows both.

​
When the engine changes

Jev current follows its provider’s model, and each change in its behaviour that we detect starts a new epoch, recorded on every answer’s engine_version. A new epoch starts with no outcomes. Until it has 100, its answers use the previous epoch’s calibration and say so with "from_previous_epoch": true. Keep posting outcomes and the new epoch gets its own fit. An exact engine version never changes, so it is one epoch for its whole life. A new judgment version starts fresh too. It needs its own 100 outcomes before its answers carry calibrated.

​
Templates and composites

  • A template judgment’s report on its prefix pools the outcomes of every namespace under it, and inherited answers use that pooled fit.
  • A composite judgment has no calibrated object. Its p already comes from weights fitted on your labels, refitted nightly in the same run.
To post outcomes, read the report and pick thresholds, see measure, improve and tune.