> ## Documentation Index > Fetch the complete documentation index at: https://docs.vainona.ai/llms.txt > Use this file to discover all available pages before exploring further. # Composite judgments > Split a fuzzy yes/no question into narrow parts, combined by weights fitted on your labels, and measured on them before it answers. export const productName = "Vainona"; A fuzzy yes/no question is sometimes answered better as several narrow ones. A composite judgment asks the engine 2 to 8 narrow yes/no questions, called parts, and combines their answers with weights fitted on your labels. It helps some judgments and not others, and nobody can tell which in advance. So {productName} measures every composite on your labels, against a fairly tuned single question, before it answers anything. ## When splitting helps We tested splitting on two human-labelled datasets. Every number below is accuracy on held-out documents, half of them positive. | | Clickbait (news posts) | Argument quality | | -------------------------------------- | ---------------------- | --------------------------- | | One question, default 0.5 cut-off | 69.7% | 61.1% | | One question, cut-off fitted on labels | 69.9% | 67.8% | | Five hand-written parts | 73.3% | 68.2% | | Parts proposed by an LLM | 75% to 81% (four sets) | 66.9% to 67.5% (three sets) | * **On clickbait, splitting helped.** Every split set beat the tuned single question. The best sets asked whether the post *states the news*, where a yes means "not clickbait", the half of the definition the single question left implicit. The one LLM-proposed set without that question scored lowest. With 50 labels, the combiner got within about 2 points of what 1,000 labels gave. * **On argument quality, it did not.** No split set did better than the tuned single question. Every part measured much the same thing as the question itself, so combining them added nothing. What helped was fitting the cut-off: it lifted the single question from 61.1% to 67.8%, a gain of 6.7 points. The split sets also needed about 300 labels just to match the single question. Splitting pays when the question hides several distinct signals that the engine can see on their own. When it does not, a composite costs more per document and does no better. ## Try the threshold recommender first On argument quality, fitting the cut-off was worth more than any split, and it is cheaper. It costs nothing per document, needs no new version, and takes one call. If your judgment has outcomes, [pick a threshold with the recommender](/guides/measure-improve-tune#pick-thresholds-with-the-recommender) first. Reach for a composite when a well-chosen threshold still gets too many documents wrong. ## Define the parts A composite is a `bool` judgment with `parts`. Post it like any other definition. On an existing name it is version n+1: ```json theme={null} { "name": "clickbait", "type": "bool", "question": "Is this post written to lure readers into clicking rather than to tell them the news?", "parts": [ {"name": "withholds_fact", "question": "Does the post hold back the key fact, so the reader must click to find out?"}, {"name": "states_fact", "question": "Does the post state a specific, concrete news fact?"} ], "context": {"fields": ["state.text"]}, "engine": {"name": "jev", "version": "current"} } ``` ```ts TypeScript theme={null} await ns.judgments.create({ name: "clickbait", type: "bool", question: "Is this post written to lure readers into clicking rather than to tell them the news?", parts: [ { name: "withholds_fact", question: "Does the post hold back the key fact, so the reader must click to find out?" }, { name: "states_fact", question: "Does the post state a specific, concrete news fact?" }, ], context: { fields: ["state.text"] }, engine: { name: "jev", version: "current" }, }); ``` ```python Python theme={null} ns.judgments.create( name="clickbait", type="bool", question="Is this post written to lure readers into clicking rather than to tell them the news?", parts=[ {"name": "withholds_fact", "question": "Does the post hold back the key fact, so the reader must click to find out?"}, {"name": "states_fact", "question": "Does the post state a specific, concrete news fact?"}, ], context={"fields": ["state.text"]}, engine={"name": "jev", "version": "current"}, ) ``` * **2 to 8 parts**, each a narrow yes/no question with a name unique within the judgment. * **Parts share the judgment's context recipe and engine.** On Jev they go in one request with the judgment's other questions, and each part counts toward the engine's limit of questions per request. * **`question` is not sent to the engine.** It documents what the combination means. The engine is asked the parts. * **Parts are part of the version.** Changing one creates a new version, like any other change to the definition. * **One level only.** A part is a question, never another judgment, so composites never nest. * **Only `bool`.** There are no composite `choice` or `score` judgments, and all parts use the same context recipe. Good parts are narrow and different from each other. Include one that points the other way, where a yes means the answer is no: on clickbait, "does it state a concrete news fact" was the strongest part in every set that had it. ## Activate it, measured on your labels A composite version is never active when it is created, not even as a judgment's first version, and `activate: true` is refused with `invalid_request`. It cannot answer until its combiner is fitted on your labels. Post [labelled examples](/guides/measure-improve-tune#post-labelled-examples) for the judgment, then activate the version with `POST /namespaces/{ns}/judgments/clickbait/activate`: ```json theme={null} {"version": 1} ``` Activation always returns `202` with a `shadow` job in `awaiting_confirm`, even with `force: true`: a composite cannot answer without a combiner, so there is nothing to skip to. The job: 1. Judges up to 1,000 documents that have outcomes for this judgment, asking each part. 2. Fits the combiner with 5-fold cross-validation, so every document is scored by a fit that never saw its label. 3. Scores a baseline the same way: the active version, or for a first version the composite's `question` asked on its own. The baseline's cut-off is fitted by the same folds on the same labels, never left at 0.5, so splitting gets no credit for what a fitted threshold alone would give. When it is done, `report` compares them. For the two-part first version of `clickbait` above: ```json theme={null} { "type": "composite", "documents": 1000, "labels": {"true": 500, "false": 500}, "composite": {"accuracy": 0.803, "accuracy_interval": {"lower": 0.777, "upper": 0.826}, "auc": 0.879}, "baseline": {"version": null, "threshold": 0.47, "accuracy": 0.699, "accuracy_interval": {"lower": 0.67, "upper": 0.727}, "auc": 0.729}, "difference": {"accuracy": 0.104, "interval": {"lower": 0.072, "upper": 0.136}}, "parts": [ {"name": "withholds_fact", "weight": 0.74}, {"name": "states_fact", "weight": -1.42} ], "verdict": "better", "recompute": {"documents": 19484, "tokens": 506584, "judgment_units": 38968, "cost_usd": 3.9, "duration_s": 1624} } ``` * **`composite`** and **`baseline`** give the cross-validated accuracy with its 95% interval, and the ROC AUC. `baseline.threshold` is the cut-off fitted for it. * **`difference`** is the composite's accuracy minus the baseline's, with its 95% interval over the same documents. * **`verdict`** is `better` only when the composite beats the baseline by more than that interval. Otherwise it is `not_better`, and the [threshold recommender](/guides/measure-improve-tune#pick-thresholds-with-the-recommender) on the baseline is the cheaper fix. * **`parts`** gives each part's weight in the combiner fitted on all the documents, on a common scale, so sizes compare. A negative weight means a yes points to `false`. A part with a weight near 0 adds cost and little else. * **`recompute`** estimates backfilling every document under the new version. Each part is billed as a judgment, so it is several times a single question's estimate. Only a composite report has `type`, so `"type" in report` tells it apart from the report of an ordinary version change: ```ts TypeScript theme={null} const activation = await ns.judgments.activate("clickbait", { version: 1 }); // A composite always returns its shadow job. Its report fills in when the sample is done. if ("id" in activation) { const { report } = await db.jobs.get(activation.id); if (report && "type" in report) console.log(report.verdict, report.difference.accuracy); } ``` ```python Python theme={null} activation = ns.judgments.activate("clickbait", version=1) # A composite always returns its shadow job. Its report fills in when the sample is done. report = db.jobs.get(activation["id"])["report"] if report is not None and "type" in report: print(report["verdict"], report["difference"]["accuracy"]) ``` Then decide. `POST /jobs/{id}/confirm` activates the version with the combiner fitted on all the labelled documents, whatever the `verdict`. `POST /jobs/{id}/cancel` leaves the active version as it is. The shadow job is free. **Labels.** The job needs at least 50 labelled documents, and at least 10 of each answer. With fewer it fails: `status` is `failed`, and `error` starts with `insufficient_labels`. Post more labels and activate again. ## Answers A composite's answer has `p`, the combined probability, and `parts`, each part's raw `p` from the engine: ```json theme={null} { "clickbait": { "type": "bool", "p": 0.87, "parts": {"withholds_fact": 0.82, "states_fact": 0.11}, "combiner": {"outcomes": 1000, "from_previous_epoch": false}, "thresholds": {"flag": true}, "freshness": "fresh", "judgment_version": 1, "engine": "jev", "engine_version": "current+2026-09-24.1", "evaluation_id": "ev_01j8zq3k5v9w2x4y6z8a0b1c2d" } } ``` * **There is no `calibrated` object.** The combiner is already fitted to your outcomes, so there is nothing left to calibrate. * **`combiner`** says what `p` came from: `outcomes`, the labelled documents it was fitted on, and `from_previous_epoch`. * **The combiner is fitted per version and per engine epoch,** and refitted nightly along with calibration. After a Jev drift, answers under the new epoch use the previous epoch's combiner, marked `"from_previous_epoch": true`, until the new epoch has 50 outcomes, at least 10 of each answer, and the nightly refit has run. * **The calibration report** measures the combined `p` per epoch. For a composite, `method`, `fitted_at` and `calibrated` are `null`, because there is nothing to calibrate. The evaluation behind an answer stores each part's raw `p`; the combined `p` is computed when the answer is read. To see what the engine returned, and how long it took, read the evaluation by its id: ```ts TypeScript theme={null} const evaluation = await ns.evaluation("ev_01j8zq3k5v9w2x4y6z8a0b1c2d"); evaluation.output?.parts; // {withholds_fact: 0.82, states_fact: 0.11} evaluation.latency_ms; ``` ```python Python theme={null} evaluation = ns.evaluation("ev_01j8zq3k5v9w2x4y6z8a0b1c2d") evaluation["output"]["parts"] # {"withholds_fact": 0.82, "states_fact": 0.11} ``` ## Thresholds, filters and ranking Everything that reads `p` reads the combined `p`: [thresholds](/concepts/judgments#thresholds), filters such as `["answers.clickbait.p", "Gte", 0.8]`, `rank_by`, and threshold recommendations. A part's `p` is in the answer for you to read, but you cannot filter or rank on it. ## Suggested parts {productName} can ask a general-purpose LLM to propose parts from your labelled examples. On clickbait, LLM-proposed sets scored between 75% and 81%, above the hand-written set's 73.3%, but on argument quality they did no better than one question. A suggestion is a starting point for a composite, not a sign that one will help. It is **off by default**, because it sends your labelled examples to an LLM provider, which is then a subprocessor, listed in the data processing agreement. An org admin turns on **Suggestions** in the organization's settings in the dashboard. Until then the call is refused with `forbidden`. `POST /namespaces/{ns}/judgments/{name}/suggest_parts` with the number of parts to propose, 2 to 8: ```json theme={null} {"count": 5} ``` ```ts TypeScript theme={null} const { parts, examples } = await ns.judgments.suggestParts("clickbait", { count: 5 }); ``` ```python Python theme={null} suggestion = ns.judgments.suggest_parts("clickbait", count=5) parts, examples = suggestion["parts"], suggestion["examples"] ``` ```json theme={null} { "parts": [ {"name": "withholds_detail", "question": "Does the post withhold a key detail (e.g., the answer, outcome, or subject's name) that requires clicking to find out?"}, {"name": "sensational_language", "question": "Does the post use sensational, emotionally charged, or superlative language?"}, {"name": "addresses_reader", "question": "Is the post phrased as a question or direct address to the reader (e.g., 'you', 'your')?"}, {"name": "vague_references", "question": "Does the post use vague or teasing references (e.g., 'this', 'these', 'someone') instead of specific facts?"}, {"name": "states_news_fact", "question": "Does the post state a specific, concrete news fact (who/what/when/where) rather than teasing a topic?"} ], "examples": ["post_804119512010424320", "post_804133678003666946", "post_804134198655221765"] } ``` * **What the LLM sees:** the judgment's `question` and `criteria` from its newest version, and up to 30 of the namespace's labelled examples for the judgment, balanced between `true` and `false` and compiled with the judgment's context recipe. `examples` lists the documents it saw. * **It never creates a version.** Review the parts, edit or drop any, then post a new version with them. `parts` has the shape a definition's `parts` takes. * **It needs labels:** at least 10 labelled examples of each answer, or it is refused with `insufficient_labels`. * **It is free,** and limited to 20 calls per judgment and 50 per organization a day (`rate_limited`, with `details.limit`, the limit reached, 20 or 50, and `details.resets_at`, the next midnight UTC). Each call shows the LLM at most 100,000 tokens of examples, so long documents are cut. When the LLM is unavailable it returns `engine_unavailable` with `Retry-After`, which the SDKs wait out and retry. A call that fails this way before the LLM ran does not count toward the day's calls; one that timed out, or failed after the LLM replied, does. In our tests, sets proposed from different examples were close paraphrases of each other. On clickbait, what separated the strong sets from the weak one was a single part, so read each set for the question it is missing. ## Pricing Each part is billed as a judgment, over the context and that part's own question. A five-part composite costs five judgment units per document per started 1,000 tokens of compiled context and part question, where a single question costs one. Features cost nothing. The shadow job that measures it is free, like every shadow job, and suggestions are free. See [pricing](/pricing#judgments).