Composite judgments

A fuzzy yes/no question is sometimes answered better as several narrow ones. A composite judgment asks the engine 2 to 8 narrow yes/no questions, called parts, and combines their answers with weights fitted on your labels. It helps some judgments and not others, and nobody can tell which in advance. So Vainona measures every composite on your labels, against a fairly tuned single question, before it answers anything.

When splitting helps

We tested splitting on two human-labelled datasets. Every number below is accuracy on held-out documents, half of them positive.

	Clickbait (news posts)	Argument quality
One question, default 0.5 cut-off	69.7%	61.1%
One question, cut-off fitted on labels	69.9%	67.8%
Five hand-written parts	73.3%	68.2%
Parts proposed by an LLM	75% to 81% (four sets)	66.9% to 67.5% (three sets)

On clickbait, splitting helped. Every split set beat the tuned single question. The best sets asked whether the post states the news, where a yes means “not clickbait”, the half of the definition the single question left implicit. The one LLM-proposed set without that question scored lowest. With 50 labels, the combiner got within about 2 points of what 1,000 labels gave.
On argument quality, it did not. No split set did better than the tuned single question. Every part measured much the same thing as the question itself, so combining them added nothing. What helped was fitting the cut-off: it lifted the single question from 61.1% to 67.8%, a gain of 6.7 points. The split sets also needed about 300 labels just to match the single question.

Splitting pays when the question hides several distinct signals that the engine can see on their own. When it does not, a composite costs more per document and does no better.

Try the threshold recommender first

On argument quality, fitting the cut-off was worth more than any split, and it is cheaper. It costs nothing per document, needs no new version, and takes one call. If your judgment has outcomes, pick a threshold with the recommender first. Reach for a composite when a well-chosen threshold still gets too many documents wrong.

Define the parts

A composite is a bool judgment with parts. Post it like any other definition. On an existing name it is version n+1:

{
  "name": "clickbait",
  "type": "bool",
  "question": "Is this post written to lure readers into clicking rather than to tell them the news?",
  "parts": [
    {"name": "withholds_fact", "question": "Does the post hold back the key fact, so the reader must click to find out?"},
    {"name": "states_fact", "question": "Does the post state a specific, concrete news fact?"}
  ],
  "context": {"fields": ["state.text"]},
  "engine": {"name": "jev", "version": "current"}
}

await ns.judgments.create({
  name: "clickbait",
  type: "bool",
  question: "Is this post written to lure readers into clicking rather than to tell them the news?",
  parts: [
    { name: "withholds_fact", question: "Does the post hold back the key fact, so the reader must click to find out?" },
    { name: "states_fact", question: "Does the post state a specific, concrete news fact?" },
  ],
  context: { fields: ["state.text"] },
  engine: { name: "jev", version: "current" },
});

ns.judgments.create(
    name="clickbait",
    type="bool",
    question="Is this post written to lure readers into clicking rather than to tell them the news?",
    parts=[
        {"name": "withholds_fact", "question": "Does the post hold back the key fact, so the reader must click to find out?"},
        {"name": "states_fact", "question": "Does the post state a specific, concrete news fact?"},
    ],
    context={"fields": ["state.text"]},
    engine={"name": "jev", "version": "current"},
)

2 to 8 parts, each a narrow yes/no question with a name unique within the judgment.
Parts share the judgment’s context recipe and engine. On Jev they go in one request with the judgment’s other questions, and each part counts toward the engine’s limit of questions per request.
question is not sent to the engine. It documents what the combination means. The engine is asked the parts.
Parts are part of the version. Changing one creates a new version, like any other change to the definition.
One level only. A part is a question, never another judgment, so composites never nest.
Only bool. There are no composite choice or score judgments, and all parts use the same context recipe.

Good parts are narrow and different from each other. Include one that points the other way, where a yes means the answer is no: on clickbait, “does it state a concrete news fact” was the strongest part in every set that had it.

Activate it, measured on your labels

A composite version is never active when it is created, not even as a judgment’s first version, and activate: true is refused with invalid_request. It cannot answer until its combiner is fitted on your labels. Post labelled examples for the judgment, then activate the version with POST /namespaces/{ns}/judgments/clickbait/activate:

{"version": 1}

Activation always returns 202 with a shadow job in awaiting_confirm, even with force: true: a composite cannot answer without a combiner, so there is nothing to skip to. The job:

Judges up to 1,000 documents that have outcomes for this judgment, asking each part.
Fits the combiner with 5-fold cross-validation, so every document is scored by a fit that never saw its label.
Scores a baseline the same way: the active version, or for a first version the composite’s question asked on its own. The baseline’s cut-off is fitted by the same folds on the same labels, never left at 0.5, so splitting gets no credit for what a fitted threshold alone would give.

When it is done, report compares them. For the two-part first version of clickbait above:

{
  "type": "composite",
  "documents": 1000,
  "labels": {"true": 500, "false": 500},
  "composite": {"accuracy": 0.803, "accuracy_interval": {"lower": 0.777, "upper": 0.826}, "auc": 0.879},
  "baseline": {"version": null, "threshold": 0.47, "accuracy": 0.699, "accuracy_interval": {"lower": 0.67, "upper": 0.727}, "auc": 0.729},
  "difference": {"accuracy": 0.104, "interval": {"lower": 0.072, "upper": 0.136}},
  "parts": [
    {"name": "withholds_fact", "weight": 0.74},
    {"name": "states_fact", "weight": -1.42}
  ],
  "verdict": "better",
  "recompute": {"documents": 19484, "tokens": 506584, "judgment_units": 38968, "cost_usd": 3.9, "duration_s": 1624}
}

composite and baseline give the cross-validated accuracy with its 95% interval, and the ROC AUC. baseline.threshold is the cut-off fitted for it.
difference is the composite’s accuracy minus the baseline’s, with its 95% interval over the same documents.
verdict is better only when the composite beats the baseline by more than that interval. Otherwise it is not_better, and the threshold recommender on the baseline is the cheaper fix.
parts gives each part’s weight in the combiner fitted on all the documents, on a common scale, so sizes compare. A negative weight means a yes points to false. A part with a weight near 0 adds cost and little else.
recompute estimates backfilling every document under the new version. Each part is billed as a judgment, so it is several times a single question’s estimate.

Only a composite report has type, so "type" in report tells it apart from the report of an ordinary version change:

const activation = await ns.judgments.activate("clickbait", { version: 1 });
// A composite always returns its shadow job. Its report fills in when the sample is done.
if ("id" in activation) {
  const { report } = await db.jobs.get(activation.id);
  if (report && "type" in report) console.log(report.verdict, report.difference.accuracy);
}

activation = ns.judgments.activate("clickbait", version=1)
# A composite always returns its shadow job. Its report fills in when the sample is done.
report = db.jobs.get(activation["id"])["report"]
if report is not None and "type" in report:
    print(report["verdict"], report["difference"]["accuracy"])

Then decide. POST /jobs/{id}/confirm activates the version with the combiner fitted on all the labelled documents, whatever the verdict. POST /jobs/{id}/cancel leaves the active version as it is. The shadow job is free. Labels. The job needs at least 50 labelled documents, and at least 10 of each answer. With fewer it fails: status is failed, and error starts with insufficient_labels. Post more labels and activate again.

Answers

A composite’s answer has p, the combined probability, and parts, each part’s raw p from the engine:

{
  "clickbait": {
    "type": "bool",
    "p": 0.87,
    "parts": {"withholds_fact": 0.82, "states_fact": 0.11},
    "combiner": {"outcomes": 1000, "from_previous_epoch": false},
    "thresholds": {"flag": true},
    "freshness": "fresh",
    "judgment_version": 1,
    "engine": "jev",
    "engine_version": "current+2026-09-24.1",
    "evaluation_id": "ev_01j8zq3k5v9w2x4y6z8a0b1c2d"
  }
}

There is no calibrated object. The combiner is already fitted to your outcomes, so there is nothing left to calibrate.
combiner says what p came from: outcomes, the labelled documents it was fitted on, and from_previous_epoch.
The combiner is fitted per version and per engine epoch, and refitted nightly along with calibration. After a Jev drift, answers under the new epoch use the previous epoch’s combiner, marked "from_previous_epoch": true, until the new epoch has 50 outcomes, at least 10 of each answer, and the nightly refit has run.
The calibration report measures the combined p per epoch. For a composite, method, fitted_at and calibrated are null, because there is nothing to calibrate.

The evaluation behind an answer stores each part’s raw p; the combined p is computed when the answer is read. To see what the engine returned, and how long it took, read the evaluation by its id:

const evaluation = await ns.evaluation("ev_01j8zq3k5v9w2x4y6z8a0b1c2d");
evaluation.output?.parts; // {withholds_fact: 0.82, states_fact: 0.11}
evaluation.latency_ms;

evaluation = ns.evaluation("ev_01j8zq3k5v9w2x4y6z8a0b1c2d")
evaluation["output"]["parts"]  # {"withholds_fact": 0.82, "states_fact": 0.11}

Thresholds, filters and ranking

Everything that reads p reads the combined p: thresholds, filters such as ["answers.clickbait.p", "Gte", 0.8], rank_by, and threshold recommendations. A part’s p is in the answer for you to read, but you cannot filter or rank on it.

Suggested parts

Vainona can ask a general-purpose LLM to propose parts from your labelled examples. On clickbait, LLM-proposed sets scored between 75% and 81%, above the hand-written set’s 73.3%, but on argument quality they did no better than one question. A suggestion is a starting point for a composite, not a sign that one will help. It is off by default, because it sends your labelled examples to an LLM provider, which is then a subprocessor, listed in the data processing agreement. An org admin turns on Suggestions in the organization’s settings in the dashboard. Until then the call is refused with forbidden. POST /namespaces/{ns}/judgments/{name}/suggest_parts with the number of parts to propose, 2 to 8:

{"count": 5}

const { parts, examples } = await ns.judgments.suggestParts("clickbait", { count: 5 });

suggestion = ns.judgments.suggest_parts("clickbait", count=5)
parts, examples = suggestion["parts"], suggestion["examples"]

{
  "parts": [
    {"name": "withholds_detail", "question": "Does the post withhold a key detail (e.g., the answer, outcome, or subject's name) that requires clicking to find out?"},
    {"name": "sensational_language", "question": "Does the post use sensational, emotionally charged, or superlative language?"},
    {"name": "addresses_reader", "question": "Is the post phrased as a question or direct address to the reader (e.g., 'you', 'your')?"},
    {"name": "vague_references", "question": "Does the post use vague or teasing references (e.g., 'this', 'these', 'someone') instead of specific facts?"},
    {"name": "states_news_fact", "question": "Does the post state a specific, concrete news fact (who/what/when/where) rather than teasing a topic?"}
  ],
  "examples": ["post_804119512010424320", "post_804133678003666946", "post_804134198655221765"]
}

What the LLM sees: the judgment’s question and criteria from its newest version, and up to 30 of the namespace’s labelled examples for the judgment, balanced between true and false and compiled with the judgment’s context recipe. examples lists the documents it saw.
It never creates a version. Review the parts, edit or drop any, then post a new version with them. parts has the shape a definition’s parts takes.
It needs labels: at least 10 labelled examples of each answer, or it is refused with insufficient_labels.
It is free, and limited to 20 calls per judgment and 50 per organization a day (rate_limited, with details.limit, the limit reached, 20 or 50, and details.resets_at, the next midnight UTC). Each call shows the LLM at most 100,000 tokens of examples, so long documents are cut. When the LLM is unavailable it returns engine_unavailable with Retry-After, which the SDKs wait out and retry. A call that fails this way before the LLM ran does not count toward the day’s calls; one that timed out, or failed after the LLM replied, does.

In our tests, sets proposed from different examples were close paraphrases of each other. On clickbait, what separated the strong sets from the weak one was a single part, so read each set for the question it is missing.

Pricing

Each part is billed as a judgment, over the context and that part’s own question. A five-part composite costs five judgment units per document per started 1,000 tokens of compiled context and part question, where a single question costs one. Features cost nothing. The shadow job that measures it is free, like every shadow job, and suggestions are free. See pricing.

Get started

Before you build

Concepts

Guides

Engines

Reference

Composite judgments

When splitting helps

Try the threshold recommender first

Define the parts

Activate it, measured on your labels

Answers

Thresholds, filters and ranking

Suggested parts

Pricing

​When splitting helps

​Try the threshold recommender first

​Define the parts

​Activate it, measured on your labels

​Answers

​Thresholds, filters and ranking

​Suggested parts

​Pricing

When splitting helps

Try the threshold recommender first

Define the parts

Activate it, measured on your labels

Answers

Thresholds, filters and ranking

Suggested parts

Pricing