Skip to main content
An evaluation is an immutable record of one computation: the document revision, judgment version, compiled context hash, engine and version, raw output, time and status. Answers point at evaluations. Evaluations are append-only on object storage and cost close to nothing to keep, so every one is kept, about 3 KB each including the compiled context. GET /namespaces/{ns}/documents/{id}?include=history returns a document’s evaluations for its current incarnation, newest first. Add all_incarnations=true for earlier lives of the same id, and history_limit to bound the list.
GET /namespaces/{ns}/evaluations/{id} (ns.evaluation(id) in both SDKs) returns one evaluation by its id, such as an answer’s evaluation_id, from any document, incarnation or judgment, shadow evaluations included. It always has context and raw, and latency_ms, how long the engine request took, retries excluded. latency_ms is absent on evaluations recorded before it was measured.
  • output holds the engine’s raw numbers: p, value, dist, escape_p or score. A composite judgment’s evaluation has parts, each part’s p; its combined p is computed when the answer is read.
  • include=history,context adds context, the exact compiled context the engine saw. include=history,raw adds raw, the engine’s raw response, without the engine’s token counts: context_tokens is the size that matters to you. Both are large, so they are left out by default.
  • context_truncated is true when the context was cut to fit max_tokens or the engine’s limit.
  • status: "failed" comes with error: {class, message}. retryable failures are retried with backoff and then hourly. terminal failures wait for the document’s next write. A document’s answer becomes a terminal failure, kept until the document is written again, in two cases: it failed for a reason of its own (the engine refused it or could not process it, such as output that is invalid or cannot be parsed, a document too large, or a content refusal), which is terminal at once; or it is still failing 24 hours after it last changed. Failures of the engine as a whole (server errors, timeouts, lost connections, rate limits) are retried: in an outage, documents are held pending. The 24-hour rule applies whatever the cause. A context the engine will not take even cut to 80% of its limit fails terminal with a message starting too_large:: lower the recipe’s max_tokens or send fewer fields.
  • The evaluation of a judgment with relations has watermark, the log position its context was read at, and related_documents: every related document the context read, rendered or aggregated, as {relation, document_id, revision}. The list is part of the context, so it comes with include=history,context and from GET /namespaces/{ns}/evaluations/{id}. It is what lets an outcome or an audit see exactly what the engine read, after the documents have changed.
  • shadow: true marks evaluations from an activation’s shadow report. They never produce answers and are not billed.
The context hash covers the compiled context, the judgment version and the engine version. When a document changes in a way that leaves a judgment’s context unchanged, the previous answer is reused without calling the engine, and no evaluation is recorded: the answer’s evaluation_id still names the earlier evaluation, for an earlier revision.