Import existing data

This guide loads a large set of existing records into Vainona: the backlog of tickets, orders, posts or conversations you already have. In short: shape each record as a document with its original created_at, and hand them all to the SDK’s import helper, which writes full batches a few at a time and retries what fails. Then judge what you imported with one backfill per judgment, after you have seen its estimate.

Pick the namespace

Import into a namespace of your own, such as acme/prod/tenant_123, not default/quickstart. The quickstart namespace holds at most 10,000 documents and 100 MB, a write past that is refused with too_large, and its judging pauses for the month at its usage limit. A namespace is created by its first write or judgment. If you serve tenants, give each its own namespace: see many tenants.

Shape each document

A document is an id, flat attributes and a JSON state (documents). Decide what goes where before you write the first batch, because moving a field later means rewriting every document.

Put it in	When	Limits
`id`	Your record’s own primary key, so a second run of the import rewrites the same documents instead of adding new ones	Up to 128 bytes of `A-Z a-z 0-9 . _ : / -`, unique in the namespace
`attributes`	Anything you filter, sort or scope on: kind, plan, region, status, and the ids of other documents a relation joins on	Strings, numbers, booleans, string arrays and nulls, at most 64 keys. Never sent to an engine unless a context recipe names them
`state`	What judgments read: subjects, bodies, messages, line items	Any JSON object up to 1 MB. Never filterable
`created_at`	The record’s original creation time, as RFC 3339	At most 5 minutes in the future. Used only when the write creates the document

Keep ids stable. If records from different tables share one namespace and their keys can collide, prefix them, such as order:1042 and product:77. An attribute that points at another document holds that document’s id exactly, prefix included.
Use a kind attribute when one namespace holds several kinds of record. Judgments use it in applies_to, and relations in match.
Send each record’s created_at. Relations read related documents newest created first, a relation’s window counts from when they were created, and so does a referenced document’s re-judge scope. With the original times, imported history reads as it happened. Without them, every document is created at the moment of its import: history sorts by import order, a window holds everything you imported, and for 30 days every imported judged document is inside the default re-judge scope.
created_at is set once. It applies only to the write that creates the document. A later write, including a retry or a second run of the import, keeps the first value and ignores the one it sends. To change it, delete the document and write it again.
Keep other dates you filter or sort on in attributes, such as when a ticket was closed.

Write with the import helper

ns.import_documents(records) in Python and ns.importDocuments(records) in TypeScript do the writing for you (SDKs). They read your records as they go, from any iterable (or async iterable in TypeScript), so the import never has to fit in memory, and:

write full batches of upserts. Each write takes up to 1,000 documents or 64 MB, whichever comes first, and commits atomically: it is written in full or not at all. Fewer, larger writes finish an import sooner than many small ones.
keep a few writes in flight, 4 by default (concurrency).
never write one id in two requests at once. If your records repeat an id, the later record waits for the write holding the earlier one, so the later record is the one that stays.
retry what is safe to retry, each batch under its own Idempotency-Key (see retry safely).
carry on past a batch that still fails, and list it with its ids and error in the summary they return. Set stop_on_failure (stopOnFailure) to stop at the first one instead.

They write with upsert, which replaces the whole document, so writing it twice leaves the same document. A patch from two writers at once can be lost in rare cases (see system behavior); an upsert cannot. They leave wait_for off: only the first 16 documents of a write are judged ahead of the rest, and an import has nothing to wait for. Writes are billed by the bytes written, and stored documents by the bytes stored: see pricing.

How many writes at once

Across the import, your API key allows 1,000 requests a second by default. Every response carries X-RateLimit-Limit, the requests a second your key allows, and X-RateLimit-Remaining, what is left in the current window. With full batches, an import rarely comes near it.
Per namespace, start with the helper’s default of 4 writes in flight. Raise concurrency while writes keep returning quickly, and lower it if 429 rate_limited responses keep coming.
On 429 rate_limited, the SDKs wait for Retry-After seconds, then send the same request again.
Never write the same document in two requests at once. The later commit wins, and with requests in flight together you cannot tell which that is. The helper sees to this within one import; do not run two imports of the same records at once.
Several namespaces can import in parallel, each with its own import, within your key’s rate limit.

Retry safely

Every write is idempotent, so retrying one is always safe, whatever went wrong.

What to retry: 429 rate_limited, 500 internal (including one whose message says the write may have been committed: it committed all of it or none of it), any other 5xx, timeouts and dropped connections. Back off between attempts.
What not to retry: 400 invalid_request and 413 too_large will fail again until you fix the request. 402 budget_exceeded needs a budget change first (see budgets). 401 unauthorized and 403 forbidden need a different key. 409 conflict means the namespace is being deleted.
What the SDKs do. They retry network errors, timeouts, 429 and 503 with a Retry-After of at most 60 seconds, and 500, 502, 503 and 504, waiting 0.5 seconds and doubling up to 8 seconds between attempts, or the Retry-After. They make 2 retries by default; for a long import, raise it on the client (max_retries in Python, maxRetries in TypeScript), as the examples do. What still fails is listed in the import’s failures.
A retried upsert writes the same document again, at a new revision. A judgment whose compiled context did not change keeps its answer, and that is not billed.
Idempotency-Key. Every write accepts this header. While the service still has the original response, a request with the same key gets that response back instead of writing again. It is best effort, and correctness never depends on it. The import helper gives each batch its own key and reuses it on that batch’s retries. For your own writes, pass idempotency_key (idempotencyKey), or send the header yourself, as in the curl example.

When to create judgments

This choice decides what the import costs and when you see the number.

Option A: judgments first

Create the judgments, then import. Every imported document is judged in the background as it lands, like any other change under on_change; wait_for would speed up only the first 16 documents of each write. The answers arrive during the import, but there is no estimate before it starts, and nothing to confirm. Only the namespace budget, or switching the judgment off on_change, stops it. It also judges some documents more than once:

A document written again during the import, such as a second pass that fills in a field, is judged again whenever its compiled context changed.
A judgment with related documents judges each judged document as its related documents arrive: once per burst, and at least once per ceiling while they keep coming.
A judgment that reads the document each judged document points at fans out whenever that document is written after the documents that point at it, to those inside its re-judge scope. With this option, import referenced documents first, such as products before order lines, and send created_at, so only recently created judged documents are in scope.

Option A suits small imports, and namespaces you fill gradually as new data arrives.

Option B: import first, then backfill

Import everything with no judgments, then create each judgment. An on_change judgment created on a namespace that already has documents judges changes from then on, and leaves the existing documents for a backfill you confirm: Vainona never backfills silently. Their answers read unavailable until the backfill reaches them. Then, for each judgment, ask for a backfill estimate:

{"estimate": {"documents": 1204332, "tokens": 2408664000, "judgment_units": 2408664, "cost_usd": 240.87, "duration_s": 107529}}

The estimate shows the documents it covers, their tokens, the judgment units and cost, and how long it will take. Nothing runs until you send it again with confirm: true, which starts a backfill job. A filters expression limits a backfill to matching documents, so you can judge the documents you need first, such as open ones, and the rest later, each with its own estimate. For a judgment with relations, the create response lists reference_index jobs in job_ids. The judgment answers once they are done, so start its backfill after. Creating one that runs on_change first returns a replay estimate of its monthly cost from your recent writes. Records imported with their original created_at count as created then, so older ones are left out of it; imported without it, the writes it replays are mostly the import itself, scaled up to a month when the namespace is younger than 30 days. The backfill estimate is the one that prices judging what you imported. So right after importing old records into a new namespace, the replay estimate can read close to zero until your own traffic arrives: ask for it again after a few days of live writes, and let the namespace budget cap spend meanwhile.

Why B for large imports

You see the cost before you pay it. Each backfill shows documents, tokens, units, cost and duration, and runs only when you confirm. Option A spends as the import runs.
The budget is checked up front. A backfill whose estimate does not fit what is left of the namespace’s monthly budget is refused with budget_exceeded when you confirm, before anything is spent. Under option A you find out when the budget pauses judging partway through.
Each document is judged once. A backfill judges what you imported as it finally is, with every related document in place, so re-writes and related documents arriving during the import cost nothing extra.
You control the pace. A backfill runs in the background, behind the namespace’s ordinary judging, and you can pause, resume or cancel it. duration_s says how long it will take, which for a large import can be days.

Choose B for any import large enough that you would want to see its cost first.

A namespace that already has judgments

To import into a namespace whose judgments run on_change, switch each one to manual with a PATCH, import, then switch it back. The switch back to on_change returns a backfill estimate for the documents with no current answer and changes nothing until you send it again with confirm: true, which switches the policy and starts the backfill. See switching to on_change. While a judgment is manual, the answers of documents that change read stale.

Set a budget first

A namespace’s budget caps what judging costs it each calendar month. Set it with PATCH /namespaces/{ns} before judging starts; the namespace must already exist, from a first write or judgment.

{"budget": {"compute_usd_per_month": 500, "on_exceeded": "pause"}}

pause stops judging when the budget is reached. Answers read stale, writes continue, and the namespace reports budget_paused: true. Raising the budget clears the pause, and on_change judgments then judge the documents that changed meanwhile.
reject makes writes fail with budget_exceeded once the budget is reached, which would stop your import partway. Use pause while you import.
Backfills are checked against the remaining budget before they start, as above. A running backfill stops at a budget pause like all judging.

See namespaces.

Watch progress

The import. The helper calls on_progress (onProgress) after each batch with the documents and batches written so far and the failures, and returns the same totals at the end. GET /namespaces/{ns} (ns.metadata()) returns stats.documents and stats.bytes, so you can compare the count with your source.
A backfill. GET /jobs/{id} (db.jobs.get(id)) returns its status, progress.documents_done, spend_usd so far, the estimate, estimated_completion_at, and error if it failed. POST /jobs/{id}/pause, /resume and /cancel control it. The dashboard shows the same jobs.
Answers. Every answer carries its freshness. For an on_change judgment, a query with "filters": ["answers.needs_escalation.freshness", "In", ["pending", "unavailable"]] and "top_k": 1 tells you whether anything is left to judge: a row, or more: true, means there is.
Spend and the budget. stats.spend_month_usd is this month’s judging cost for the namespace, and budget_paused says whether the budget has paused it.

Many tenants

One namespace per tenant, such as acme/prod/tenant_123. Each has its own documents, answers, budget and bill, and tenants import in parallel. See multi-tenant platforms.
Define judgments once, on a prefix. A judgment created on acme/prod/* is a template: every namespace under the prefix inherits it, including tenants created later.
Moving many existing tenants at once, import them all first, then create the template judgments. A namespace that existed before the template judgment leaves its documents for a backfill, and one backfill on the prefix path covers every tenant. Its estimate counts up to 100 of the namespaces and scales to all of them. Once confirmed, each namespace stays within its own budget, and one whose budget is paused is skipped.
A tenant created after the template is judged in full from its first write, as in option A. Set its budget after its first write, before the rest of its import.
URL encoding. In URL paths, send each / in a namespace name as %2F, so acme/prod/tenant_123 is acme%2Fprod%2Ftenant_123, and a template prefix is acme%2Fprod%2F*, with the * as it is. The SDKs do this for you. The same holds for document ids that contain /.

Examples

These import records(), a generator that reads your own records and shapes them as documents, into one namespace with 4 writes in flight, then estimate and confirm a backfill.

import { Client, type UpsertDocument } from "vainona";

const db = new Client({ apiKey: process.env.VAINONA_API_KEY!, maxRetries: 8 });
const ns = db.namespace("acme/prod/tenant_123");

// Your records as documents, with their original creation time.
async function* records(): AsyncGenerator<UpsertDocument> {
  for await (const ticket of readTickets()) {
    yield {
      id: `ticket:${ticket.id}`,
      attributes: { kind: "ticket", account_id: `account:${ticket.accountId}` },
      state: { subject: ticket.subject, body: ticket.body },
      created_at: ticket.createdAt.toISOString(),
    };
  }
}

const summary = await ns.importDocuments(records(), {
  concurrency: 4,
  onProgress: ({ documents, failures }) => console.log(`${documents} written, ${failures.length} batches failed`),
});
for (const { ids, error } of summary.failures) console.error(ids.length, "documents not written:", error);

// Option B: once the import is done, estimate the backfill, then confirm it.
const result = await ns.judgments.backfill("needs_escalation");
if ("estimate" in result) console.log(result.estimate);
const started = await ns.judgments.backfill("needs_escalation", { confirm: true });
if ("job_id" in started) console.log(await db.jobs.get(started.job_id));

import os
from collections.abc import Iterator

from vainona import Client
from vainona.types import UpsertDocument

db = Client(api_key=os.environ["VAINONA_API_KEY"], max_retries=8)
ns = db.namespace("acme/prod/tenant_123")


def records() -> Iterator[UpsertDocument]:
    """Your records as documents, with their original creation time."""
    for ticket in read_tickets():
        yield {
            "id": f"ticket:{ticket.id}",
            "attributes": {"kind": "ticket", "account_id": f"account:{ticket.account_id}"},
            "state": {"subject": ticket.subject, "body": ticket.body},
            "created_at": ticket.created_at.isoformat(),
        }


summary = ns.import_documents(
    records(),
    concurrency=4,
    on_progress=lambda p: print(f"{p.documents} written, {len(p.failures)} batches failed"),
)
for failure in summary.failures:
    print(len(failure.ids), "documents not written:", failure.error)

# Option B: once the import is done, estimate the backfill, then confirm it.
print(ns.judgments.backfill("needs_escalation"))
started = ns.judgments.backfill("needs_escalation", confirm=True)
if "job_id" in started:
    print(db.jobs.get(started["job_id"]))

readTickets and read_tickets stand for your own reader. In Python, ticket.created_at must carry its time zone, such as a datetime in UTC, so isoformat() gives RFC 3339. A write commits all of its documents or none, and writing them again is safe: fix what made a batch fail, then import its records again.

curl

API_BASE below is https://api.vainona.ai/v1. batch-000001.json holds one write body, such as {"upsert": [{"id": "ticket:1042", "state": {...}, "created_at": "2024-03-18T09:12:00Z"}]}. -i prints the response headers, including X-RateLimit-Remaining, and Retry-After on a 429.

# One batch, with an Idempotency-Key that is the same on every retry of this batch.
curl -i -X POST "$API_BASE/namespaces/acme%2Fprod%2Ftenant_123" \
  -H "Authorization: Bearer $VAINONA_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: import-batch-000001" \
  --data-binary @batch-000001.json

# After the import: the backfill estimate, then the same request with confirm.
curl -X POST "$API_BASE/namespaces/acme%2Fprod%2Ftenant_123/judgments/needs_escalation/backfill" \
  -H "Authorization: Bearer $VAINONA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"confirm": false}'

curl -X POST "$API_BASE/namespaces/acme%2Fprod%2Ftenant_123/judgments/needs_escalation/backfill" \
  -H "Authorization: Bearer $VAINONA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"confirm": true}'

# Follow the job it returns.
curl "$API_BASE/jobs/$JOB_ID" -H "Authorization: Bearer $VAINONA_API_KEY"

Get started

Before you build

Concepts

Guides

Engines

Reference

Import existing data

Pick the namespace

Shape each document

Write with the import helper

How many writes at once

Retry safely

When to create judgments

Option A: judgments first

Option B: import first, then backfill

Why B for large imports

A namespace that already has judgments

Set a budget first

Watch progress

Many tenants

Examples

curl

Checklist

​Pick the namespace

​Shape each document

​Write with the import helper

​How many writes at once

​Retry safely

​When to create judgments

​Option A: judgments first

​Option B: import first, then backfill

​Why B for large imports

​A namespace that already has judgments

​Set a budget first

​Watch progress

​Many tenants

​Examples

​curl

​Checklist

Pick the namespace

Shape each document

Write with the import helper

How many writes at once

Retry safely

When to create judgments

Option A: judgments first

Option B: import first, then backfill

Why B for large imports

A namespace that already has judgments

Set a budget first

Watch progress

Many tenants

Examples

curl

Checklist