Judge a document with its related documents

A judgment usually reads one document. It can also read the documents that point at the one it judges: an account with its tickets and invoices, a user with their posts, a conversation with its messages. The answer stays current as any of those documents change, and each evaluation records exactly which ones it read. Without it, you copy the related records into the judged document yourself and rewrite it whenever one of them changes. With it, Vainona tracks which judged documents each write touches, waits for bursts of writes to settle, and judges each one once per burst. This guide covers documents that point at the judged one. A judgment can also read the one document the judged document points at, such as the product an order line names; see judge a document with the document it points at. Relations compares the two.

Start with the simplest context

Before you add related documents, decide what the question depends on. Try these in order:

A few recent records, as raw text. Use this when the answer is in what was said: an angry reply, a mention of a competitor, a question nobody answered. Keep last_n small, around 5 to 10, and show only the fields that carry meaning.
Aggregates. Use these when the answer depends on how much or how often: how many tickets, the total of overdue invoices, the status of the latest one. An aggregate costs a few tokens where raw text costs hundreds.
Both, as two relations on the same documents. Show the last few as text and count the last 90 days.

Nothing else is needed for most questions. Child judgments, where each related record is judged first and those answers feed the parent, are not available. The experiments below show why: they cost many times more and did not beat raw text.

What we measured

We tested the idea on two public datasets with real outcomes, fitting on 1,000 items and scoring on 1,000 held-out ones. The numbers are AUROC: 0.5 is chance and 1.0 is perfect.

What the judgment read	Will a new user still be active in 3 to 6 months? (Q&A site)	Will the next reply in a conversation be uncivil? (discussion forum)
The record alone (the user, or the conversation’s opening)	0.50	0.62
Only the most recent related record	–	0.73
The record plus its last 8 related records, as raw text	0.59	0.75
Counts of the related records, with no model	0.67	0.61
Counts combined with the judgment over the last 8 records	0.67	0.75
One line per related record with 3 child judgments, and no text	0.64	0.72

What this shows, and what it does not:

Related records beat the record alone in both tests, by 0.09 and by 0.13, and both gains were statistically significant. This is what reading related documents is for.
What matters depends on the question. For whether a user stays active, counts did best, and the text added almost nothing: people who answer others’ questions in their first month stay. For whether a reply turns uncivil, the text did best, and counts were weak. So a relation can show text, numbers or both, and you should try the one that fits your question.
Recency carries most of the signal. In the second test the most recent record alone scored 0.73 against 0.75 for the last eight, and that difference is within the noise. Newest first with a small last_n is the right default.
Counts plus a judgment was the best result, or tied with it, in both tests, and adding counts never hurt. A composite judgment can take a relation’s aggregates as features beside a judgment’s answer.
Child judgments are on hold. They carried real signal in the second test, but they did not beat raw text that fit in the context, and they cost 11.6 to 16 judgment units per item against 1 for raw text, because every child judgment is billed. They would help only when the related records are too long to send as text.
Your data decides. These are two datasets, each balanced to half positive, with one engine and one wording of each question. Measure your own judgment against your outcomes before you rely on it (see measure, improve and tune).

In both tests the context of the record plus its last 8 related records was about 430 tokens at the median, so with the question under 1,000 tokens: one judgment unit.

Define one

Related documents point at the judged one through an ordinary attribute. In this example the judged documents are accounts, and each ticket and invoice carries attributes.account_id. The judgment is an ordinary judgment with related in its context recipe, usually with applies_to so it judges only accounts:

{
  "name": "churn_risk",
  "type": "bool",
  "applies_to": {"attributes.kind": "account"},
  "question": "Is this account at risk of cancelling in the next 60 days?",
  "context": {
    "fields": ["state.name", "attributes.plan", "state.renewal_date"],
    "related": {
      "tickets": {
        "match": {"attributes.kind": "ticket"},
        "join": {"theirs": "attributes.account_id", "mine": "id"},
        "last_n": 8,
        "fields": ["created_at", "state.subject", "state.status"]
      },
      "invoices": {
        "match": {"attributes.kind": "invoice"},
        "join": {"theirs": "attributes.account_id", "mine": "id"},
        "window": "180d",
        "aggregate": {"count": true, "sum": ["state.amount"], "latest": ["state.status"]}
      }
    }
  },
  "engine": {"name": "jev", "version": "current"},
  "freshness": {"policy": "on_change", "debounce_ms": 600000, "max_wait_ms": 3600000},
  "horizon": "60d"
}

The same shape fits a user and their posts (match on posts, theirs: "attributes.user_id") or a conversation and its messages (theirs: "attributes.conversation_id").

applies_to limits which documents the judgment judges, answers and bills. Other documents have no answer for it: answers leaves it out, and filters treat it as missing.
match picks which documents a relation reads, and join says how they point at the judged one: a document belongs to the judged document whose id equals its theirs attribute.
last_n, window or both bound each relation, so a context cannot grow without limit. A relation reads at most the newest 1,000 documents either way.
fields are the paths shown from each document, and aggregate adds numbers over the same documents. Each relation needs one or both.

The context recipe guide describes every key, how the context is rendered, and the limits.

// churnRisk is the definition above.
const result = await ns.judgments.create({ ...churnRisk });
if ("replay" in result) {
  // Nothing was created. Show the estimate, then create again with confirm: true.
  console.log(result.replay);
  await ns.judgments.create({ ...churnRisk, confirm: true });
}

# churn_risk is the definition above.
result = ns.judgments.create(**churn_risk)
if "replay" in result:
    # Nothing was created. Show the estimate, then create again with confirm=True.
    print(result["replay"])
    ns.judgments.create(**churn_risk, confirm=True)

The first version that joins on a new attribute builds a reference index for it, as a reference_index job, and the create response lists it in job_ids, one job per new attribute. Each job’s attribute names the attribute it indexes. The judgment answers once the job is done; until then its answers are unavailable. A namespace has at most 3 reference indexes, one per attribute its relations join on, in either direction. Reuse an attribute when you can: a create that would need a fourth is refused, and the error names the three you have.

Newest created first, and what an edit costs

A relation orders its documents by created_at, newest first. window keeps those created within the window, and last_n then keeps the newest n. So “the last 8 tickets” means the 8 most recently created. The window counts back from the later of the judged document’s own newest write and its newest related document’s creation, not from the clock. Editing a related document moves neither the list nor the window. created_at is when a document was first written to Vainona, unless the write that created it gave its own created_at. When you import existing data, send each record’s original creation time as created_at, so relations read your history in the order it happened, and a window counts it from when it happened. A write to a related document always marks the judged document it points at as touched, but it only costs an evaluation when it changes what the engine would see:

A new related document changes the list, so the judged document is judged again, once its writes settle.
An edit to one of the last 8, in a field the relation shows, changes the context, so it is judged again.
An edit to an older one, or to a field no relation shows, leaves the compiled context exactly as it was. The answer is kept, and nothing is billed.

The tradeoff is that activity on an old related document goes unseen. If that matters, record it as a new document, such as an event, rather than an edit to the old one. Aggregates count the same selection, so they follow the same rule. To show a few documents but count many, use two relations on the same match: one with a small last_n and fields, one with a window and aggregate.

Debounce and its ceiling

A judged document with busy related documents, such as an account whose tickets keep arriving, would be judged on every write without a debounce. debounce_ms makes Vainona wait until its writes have been quiet that long, so a burst of 100 tickets in five minutes costs one evaluation. A debounce alone never ends for a judged document that gets a new related document every few minutes. The ceiling, max_wait_ms, judges it anyway once that long has passed since its oldest unjudged write. With a 10-minute debounce and a 1-hour ceiling, a ticket every minute for 3 hours costs 3 evaluations, not 180 and not 0.

max_wait_ms defaults to 12 × debounce_ms for a judgment with related documents and a debounce_ms above 0. With no debounce, and for a judgment of a single document, it defaults to no ceiling.
It must be at least debounce_ms, and null turns it off.
Like the debounce, it is a setting: change it with a PATCH and no new version. GET returns the value in effect.

Answers therefore lag their related documents by the debounce, typically minutes, and by at most the ceiling. That is the price of judging an account once per burst of tickets. A question about a single event, such as whether one payment is fraudulent, should stay a judgment of that document.

The replay estimate

The cost of a judgment with related documents depends on how often those documents change, not on how many judged documents you have. So before one runs on_change, Vainona replays your namespace’s last 30 days of writes through its relations, debounce and ceiling, and tells you what it would have cost. Two requests return this estimate and create nothing until you send confirm: true: creating a version with related on a judgment that runs on_change, and switching such a judgment to on_change.

{"replay": {"replayed_days": 30, "entities": 1000000, "judgments_per_month": 8000000, "judgment_units_per_month": 8000000, "cost_usd_per_month": 800.0, "bulk_pool_share": 0.22, "lower_bound": false}}

entities is how many documents the judgment applies to now.
judgments_per_month is how many evaluations the replay counted, and judgment_units_per_month the units they would bill, from the mean context and question of up to 1,000 of your judged documents.
cost_usd_per_month prices those units at the tiers your organization would be in, counting what it has already used this month.
bulk_pool_share is the share of the background judging rate available to the judgment that those evaluations would use. Above 1, the judgment cannot keep up with your writes: raise the debounce or the ceiling, or narrow the relations.
replayed_days is below 30 in a younger namespace, and the monthly figures are scaled up from it. A document counts as created at its created_at, so records imported with their original creation time are left out when they are older than the days replayed. Imported without it, they count as created during the import, and the replay describes the import rather than your ongoing writes.

“At least.” Storage keeps each document’s newest version, not every write. So the replay sees when each related document was created and when it was last written, but not the edits in between. For documents written once, such as events, messages and invoices, it is exact. When a related document was edited more often than the replay could see, lower_bound is true, and the dashboard shows the figures as “at least”. The replay also does not count the evaluations that unchanged contexts save, which only lower the bill. The namespace budget still caps real spend. A confirmed request whose cost_usd_per_month is more than the namespace’s monthly budget is refused with budget_exceeded, and the dashboard says so before you confirm.

What each answer and evaluation shows

An answer of a judgment with related documents carries a watermark: the position in the namespace’s log that its context was read at. Every write at or below it, to the judged document or to any document that points at it, is reflected in the answer.

A write to a related document makes the judged document it points at pending at the next read. It is fresh again once an answer lands with a watermark at or above that write.
revision still names the judged document’s own revision.
wait_for and wait_ms wait for the written or read document’s own answer. Writing a ticket with wait_for: ["churn_risk"] returns at once, because churn_risk does not apply to tickets.

Each evaluation also lists related_documents: every related document its context read, rendered or aggregated, with the relation and the revision it was read at. It comes back with include=history,context and from GET /namespaces/{ns}/evaluations/{id}. That list is what makes an audit exact after the documents have changed, and what an outcome with a horizon joins to: “this account churned” joins the evaluation that was current 60 days before, and that evaluation names the tickets it read. The dashboard’s evaluation page shows them grouped by relation.

Keep it affordable

A few habits keep the bill small, and most of them also make the answers better:

Keep last_n small. Recency carried most of the signal in both tests. The last 5 to 10 records often cost one unit; a year of history can cost ten.
Prefer aggregates when volume is the signal. A count is a few tokens.
Show only the fields that matter. Leave long bodies out of a relation unless the question depends on them.
Share recipes. Judgments with the same context recipe, such as churn_risk and expansion on the same relations, go to the engine in one request. Each is still billed, but they take fewer engine requests, so the bulk pool share stays low.
Pick the debounce for how fresh the answer must be, not shorter. A 10-minute debounce with a 1-hour ceiling judges a busy account at most a few times an hour.
Read the replay estimate. It is your own traffic, so it is the best guide to what a change will cost.

The document each judged document points at

A relation can also run the other way, and read the one document the judged document points at through its own attribute, such as an order line reading its product. A change to that document re-judges the documents that point at it, which has its own costs and controls. See judge a document with the document it points at.

Not available yet

Child judgments as context: feeding each related document’s own answers into the parent.
Joins where documents share a key, with an attribute on both sides.
Oldest-first ordering, such as the opening messages of a thread.
Named collections. Use applies_to on an attribute such as kind.
Judgments that read the answers of a judgment with related documents.

Get started

Before you build

Concepts

Guides

Engines

Reference

Judge a document with its related documents

Start with the simplest context

What we measured

Define one

Newest created first, and what an edit costs

Debounce and its ceiling

The replay estimate

What each answer and evaluation shows

Keep it affordable

The document each judged document points at

Not available yet

​Start with the simplest context

​What we measured

​Define one

​Newest created first, and what an edit costs

​Debounce and its ceiling

​The replay estimate

​What each answer and evaluation shows

​Keep it affordable

​The document each judged document points at

​Not available yet

Start with the simplest context

What we measured

Define one

Newest created first, and what an edit costs

Debounce and its ceiling

The replay estimate

What each answer and evaluation shows

Keep it affordable

The document each judged document points at

Not available yet