> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vainona.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Import existing data

> Load your existing records in full batches, retry freely, and judge them once with the cost in front of you.

export const productName = "Vainona";

export const apiBase = "https://api.vainona.ai/v1";

This guide loads a large set of existing records into {productName}: the backlog of tickets, orders, posts or conversations you already have. In short: shape each record as a document with its original `created_at`, and hand them all to the SDK's import helper, which writes full batches a few at a time and retries what fails. Then judge what you imported with one backfill per judgment, after you have seen its estimate.

## Pick the namespace

Import into a namespace of your own, such as `acme/prod/tenant_123`, not `default/quickstart`. The quickstart namespace holds at most 10,000 documents and 100 MB, a write past that is refused with `too_large`, and its judging pauses for the month at [its usage limit](/pricing#quickstart).

A namespace is created by its first write or judgment. If you serve tenants, give each its own namespace: see [many tenants](#many-tenants).

## Shape each document

A document is an `id`, flat `attributes` and a JSON `state` ([documents](/concepts/documents)). Decide what goes where before you write the first batch, because moving a field later means rewriting every document.

| Put it in    | When                                                                                                                                         | Limits                                                                                                                           |
| ------------ | -------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| `id`         | Your record's own primary key, so a second run of the import rewrites the same documents instead of adding new ones                          | Up to 128 bytes of `A-Z a-z 0-9 . _ : / -`, unique in the namespace                                                              |
| `attributes` | Anything you filter, sort or scope on: kind, plan, region, status, and the ids of other documents a [relation](/concepts/relations) joins on | Strings, numbers, booleans, string arrays and nulls, at most 64 keys. Never sent to an engine unless a context recipe names them |
| `state`      | What judgments read: subjects, bodies, messages, line items                                                                                  | Any JSON object up to 1 MB. Never filterable                                                                                     |
| `created_at` | The record's original creation time, as RFC 3339                                                                                             | At most 5 minutes in the future. Used only when the write creates the document                                                   |

* **Keep ids stable.** If records from different tables share one namespace and their keys can collide, prefix them, such as `order:1042` and `product:77`. An attribute that points at another document holds that document's id exactly, prefix included.
* **Use a `kind` attribute** when one namespace holds several kinds of record. Judgments use it in `applies_to`, and relations in `match`.
* **Send each record's `created_at`.** Relations read related documents newest created first, a relation's `window` counts from when they were created, and so does a [referenced document's](/guides/referenced-document) re-judge scope. With the original times, imported history reads as it happened. Without them, every document is created at the moment of its import: history sorts by import order, a `window` holds everything you imported, and for 30 days every imported judged document is inside the default re-judge scope.
* **`created_at` is set once.** It applies only to the write that creates the document. A later write, including a retry or a second run of the import, keeps the first value and ignores the one it sends. To change it, delete the document and write it again.
* **Keep other dates you filter or sort on in `attributes`,** such as when a ticket was closed.

## Write with the import helper

`ns.import_documents(records)` in Python and `ns.importDocuments(records)` in TypeScript do the writing for you ([SDKs](/sdks#importing-documents)). They read your records as they go, from any iterable (or async iterable in TypeScript), so the import never has to fit in memory, and:

* **write full batches of upserts.** Each write takes up to 1,000 documents or 64 MB, whichever comes first, and commits atomically: it is written in full or not at all. Fewer, larger writes finish an import sooner than many small ones.
* **keep a few writes in flight,** 4 by default (`concurrency`).
* **never write one id in two requests at once.** If your records repeat an id, the later record waits for the write holding the earlier one, so the later record is the one that stays.
* **retry what is safe to retry,** each batch under its own `Idempotency-Key` (see [retry safely](#retry-safely)).
* **carry on past a batch that still fails,** and list it with its ids and error in the summary they return. Set `stop_on_failure` (`stopOnFailure`) to stop at the first one instead.

They write with `upsert`, which replaces the whole document, so writing it twice leaves the same document. They leave `wait_for` off: only the first 16 documents of a write are judged ahead of the rest, and an import has nothing to wait for.

Writes are billed by the bytes written, and stored documents by the bytes stored: see [pricing](/pricing#usage).

## How many writes at once

* **Across the import,** your API key allows 1,000 requests a second by default. Every response carries `X-RateLimit-Limit`, the requests a second your key allows, and `X-RateLimit-Remaining`, what is left in the current window. With full batches, an import rarely comes near it.
* **Per namespace,** start with the helper's default of 4 writes in flight. Raise `concurrency` while writes keep returning quickly, and lower it if `429 rate_limited` responses keep coming.
* **On `429 rate_limited`,** the SDKs wait for `Retry-After` seconds, then send the same request again.
* **Never write the same document in two requests at once.** The later commit wins, and with requests in flight together you cannot tell which that is. The helper sees to this within one import; do not run two imports of the same records at once.
* **Several namespaces can import in parallel,** each with its own import, within your key's rate limit.

## Retry safely

Every write is idempotent, so retrying one is always safe, whatever went wrong.

* **What to retry:** `429 rate_limited`, `500 internal` (including one whose message says the write may have been committed: it committed all of it or none of it), any other `5xx`, timeouts and dropped connections. Back off between attempts.
* **What not to retry:** `400 invalid_request` and `413 too_large` will fail again until you fix the request. `402 budget_exceeded` needs a budget change first (see [budgets](#set-a-budget-first)). `401 unauthorized` and `403 forbidden` need a different key. `409 conflict` means the namespace is being deleted.
* **What the SDKs do.** They retry network errors, timeouts, `429` and `503` with a `Retry-After` of at most 60 seconds, and `500`, `502`, `503` and `504`, waiting 0.5 seconds and doubling up to 8 seconds between attempts, or the `Retry-After`. They make 2 retries by default; for a long import, raise it on the client (`max_retries` in Python, `maxRetries` in TypeScript), as the [examples](#examples) do. What still fails is listed in the import's `failures`.
* **A retried upsert** writes the same document again, at a new revision. A judgment whose compiled context did not change keeps its answer, and that is not billed.
* **`Idempotency-Key`.** Every write accepts this header. While the service still has the original response, a request with the same key gets that response back instead of writing again. It is best effort, and correctness never depends on it. The import helper gives each batch its own key and reuses it on that batch's retries. For your own writes, pass `idempotency_key` (`idempotencyKey`), or send the header yourself, as in the [curl example](#curl).

## When to create judgments

This choice decides what the import costs and when you see the number.

### Option A: judgments first

Create the judgments, then import. Every imported document is judged in the background as it lands, like any other change under `on_change`; `wait_for` would speed up only the first 16 documents of each write. The answers arrive during the import, but there is no estimate before it starts, and nothing to confirm. Only the namespace budget, or switching the judgment off `on_change`, stops it.

It also judges some documents more than once:

* A document written again during the import, such as a second pass that fills in a field, is judged again whenever its compiled context changed.
* A judgment with [related documents](/guides/related-documents) judges each judged document as its related documents arrive: once per burst, and at least once per ceiling while they keep coming.
* A judgment that reads [the document each judged document points at](/guides/referenced-document) fans out whenever that document is written after the documents that point at it, to those inside its re-judge scope. With this option, import referenced documents first, such as products before order lines, and send `created_at`, so only recently created judged documents are in scope.

Option A suits small imports, and namespaces you fill gradually as new data arrives.

### Option B: import first, then backfill

Import everything with no judgments, then create each judgment. An `on_change` judgment created on a namespace that already has documents judges changes from then on, and leaves the existing documents for a backfill you confirm: {productName} never backfills silently. Their answers read `unavailable` until the backfill reaches them.

Then, for each judgment, ask for a backfill estimate:

```json theme={null}
{"estimate": {"documents": 1204332, "tokens": 2408664000, "judgment_units": 2408664, "cost_usd": 240.87, "duration_s": 107529}}
```

The estimate shows the documents it covers, their tokens, the [judgment units](/pricing#judgments) and cost, and how long it will take. Nothing runs until you send it again with `confirm: true`, which starts a `backfill` job. A `filters` expression limits a backfill to matching documents, so you can judge the documents you need first, such as open ones, and the rest later, each with its own estimate.

For a judgment with relations, the create response lists `reference_index` jobs in `job_ids`. The judgment answers once they are done, so start its backfill after. Creating one that runs `on_change` first returns a [replay estimate](/guides/related-documents#the-replay-estimate) of its monthly cost from your recent writes. Records imported with their original `created_at` count as created then, so older ones are left out of it; imported without it, the writes it replays are mostly the import itself, scaled up to a month when the namespace is younger than 30 days. The backfill estimate is the one that prices judging what you imported. So right after importing old records into a new namespace, the replay estimate can read close to zero until your own traffic arrives: ask for it again after a few days of live writes, and let the namespace [budget](#set-a-budget-first) cap spend meanwhile.

### Why B for large imports

* **You see the cost before you pay it.** Each backfill shows documents, tokens, units, cost and duration, and runs only when you confirm. Option A spends as the import runs.
* **The budget is checked up front.** A backfill whose estimate does not fit what is left of the namespace's monthly budget is refused with `budget_exceeded` when you confirm, before anything is spent. Under option A you find out when the budget pauses judging partway through.
* **Each document is judged once.** A backfill judges what you imported as it finally is, with every related document in place, so re-writes and related documents arriving during the import cost nothing extra.
* **You control the pace.** A backfill runs in the background, behind the namespace's ordinary judging, and you can pause, resume or cancel it. `duration_s` says how long it will take, which for a large import can be days.

Choose B for any import large enough that you would want to see its cost first.

### A namespace that already has judgments

To import into a namespace whose judgments run `on_change`, switch each one to `manual` with a `PATCH`, import, then switch it back. The switch back to `on_change` returns a backfill estimate for the documents with no current answer and changes nothing until you send it again with `confirm: true`, which switches the policy and starts the backfill. See [switching to `on_change`](/guides/freshness-policies#switching-to-on_change). While a judgment is `manual`, the answers of documents that change read `stale`.

## Set a budget first

A namespace's budget caps what judging costs it each calendar month. Set it with `PATCH /namespaces/{ns}` before judging starts; the namespace must already exist, from a first write or judgment.

```json theme={null}
{"budget": {"compute_usd_per_month": 500, "on_exceeded": "pause"}}
```

* **`pause`** stops judging when the budget is reached. Answers read `stale`, writes continue, and the namespace reports `budget_paused: true`. Raising the budget clears the pause, and `on_change` judgments then judge the documents that changed meanwhile.
* **`reject`** makes writes fail with `budget_exceeded` once the budget is reached, which would stop your import partway. Use `pause` while you import.
* **Backfills are checked against the remaining budget** before they start, as above. A running backfill stops at a budget pause like all judging.

See [namespaces](/concepts/namespaces#settings).

## Watch progress

* **The import.** The helper calls `on_progress` (`onProgress`) after each batch with the documents and batches written so far and the failures, and returns the same totals at the end. `GET /namespaces/{ns}` (`ns.metadata()`) returns `stats.documents` and `stats.bytes`, so you can compare the count with your source.
* **A backfill.** `GET /jobs/{id}` (`db.jobs.get(id)`) returns its `status`, `progress.documents_done`, `spend_usd` so far, the `estimate`, `estimated_completion_at`, and `error` if it failed. `POST /jobs/{id}/pause`, `/resume` and `/cancel` control it. The dashboard shows the same jobs.
* **Answers.** Every answer carries its [freshness](/concepts/freshness). For an `on_change` judgment, a query with `"filters": ["answers.needs_escalation.freshness", "In", ["pending", "unavailable"]]` and `"top_k": 1` tells you whether anything is left to judge: a row, or `more: true`, means there is.
* **Spend and the budget.** `stats.spend_month_usd` is this month's judging cost for the namespace, and `budget_paused` says whether the budget has paused it.

## Many tenants

* **One namespace per tenant,** such as `acme/prod/tenant_123`. Each has its own documents, answers, budget and bill, and tenants import in parallel. See [multi-tenant platforms](/guides/multi-tenant-platforms).
* **Define judgments once, on a prefix.** A judgment created on `acme/prod/*` is a [template](/guides/templates): every namespace under the prefix inherits it, including tenants created later.
* **Moving many existing tenants at once,** import them all first, then create the template judgments. A namespace that existed before the template judgment leaves its documents for a backfill, and one backfill on the prefix path covers every tenant. Its estimate counts up to 100 of the namespaces and scales to all of them. Once confirmed, each namespace stays within its own budget, and one whose budget is paused is skipped.
* **A tenant created after the template** is judged in full from its first write, as in option A. Set its budget after its first write, before the rest of its import.
* **URL encoding.** In URL paths, send each `/` in a namespace name as `%2F`, so `acme/prod/tenant_123` is `acme%2Fprod%2Ftenant_123`, and a template prefix is `acme%2Fprod%2F*`, with the `*` as it is. The SDKs do this for you. The same holds for document ids that contain `/`.

## Examples

These import `records()`, a generator that reads your own records and shapes them as documents, into one namespace with 4 writes in flight, then estimate and confirm a backfill.

<CodeGroup>
  ```ts TypeScript theme={null}
  import { Client, type UpsertDocument } from "vainona";

  const db = new Client({ apiKey: process.env.VAINONA_API_KEY!, maxRetries: 8 });
  const ns = db.namespace("acme/prod/tenant_123");

  // Your records as documents, with their original creation time.
  async function* records(): AsyncGenerator<UpsertDocument> {
    for await (const ticket of readTickets()) {
      yield {
        id: `ticket:${ticket.id}`,
        attributes: { kind: "ticket", account_id: `account:${ticket.accountId}` },
        state: { subject: ticket.subject, body: ticket.body },
        created_at: ticket.createdAt.toISOString(),
      };
    }
  }

  const summary = await ns.importDocuments(records(), {
    concurrency: 4,
    onProgress: ({ documents, failures }) => console.log(`${documents} written, ${failures.length} batches failed`),
  });
  for (const { ids, error } of summary.failures) console.error(ids.length, "documents not written:", error);

  // Option B: once the import is done, estimate the backfill, then confirm it.
  const result = await ns.judgments.backfill("needs_escalation");
  if ("estimate" in result) console.log(result.estimate);
  const started = await ns.judgments.backfill("needs_escalation", { confirm: true });
  if ("job_id" in started) console.log(await db.jobs.get(started.job_id));
  ```

  ```python Python theme={null}
  import os
  from collections.abc import Iterator

  from vainona import Client
  from vainona.types import UpsertDocument

  db = Client(api_key=os.environ["VAINONA_API_KEY"], max_retries=8)
  ns = db.namespace("acme/prod/tenant_123")


  def records() -> Iterator[UpsertDocument]:
      """Your records as documents, with their original creation time."""
      for ticket in read_tickets():
          yield {
              "id": f"ticket:{ticket.id}",
              "attributes": {"kind": "ticket", "account_id": f"account:{ticket.account_id}"},
              "state": {"subject": ticket.subject, "body": ticket.body},
              "created_at": ticket.created_at.isoformat(),
          }


  summary = ns.import_documents(
      records(),
      concurrency=4,
      on_progress=lambda p: print(f"{p.documents} written, {len(p.failures)} batches failed"),
  )
  for failure in summary.failures:
      print(len(failure.ids), "documents not written:", failure.error)

  # Option B: once the import is done, estimate the backfill, then confirm it.
  print(ns.judgments.backfill("needs_escalation"))
  started = ns.judgments.backfill("needs_escalation", confirm=True)
  if "job_id" in started:
      print(db.jobs.get(started["job_id"]))
  ```
</CodeGroup>

`readTickets` and `read_tickets` stand for your own reader. In Python, `ticket.created_at` must carry its time zone, such as a `datetime` in UTC, so `isoformat()` gives RFC 3339. A write commits all of its documents or none, and writing them again is safe: fix what made a batch fail, then import its records again.

### curl

`API_BASE` below is <code>{apiBase}</code>. `batch-000001.json` holds one write body, such as `{"upsert": [{"id": "ticket:1042", "state": {...}, "created_at": "2024-03-18T09:12:00Z"}]}`. `-i` prints the response headers, including `X-RateLimit-Remaining`, and `Retry-After` on a `429`.

```sh theme={null}
# One batch, with an Idempotency-Key that is the same on every retry of this batch.
curl -i -X POST "$API_BASE/namespaces/acme%2Fprod%2Ftenant_123" \
  -H "Authorization: Bearer $VAINONA_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: import-batch-000001" \
  --data-binary @batch-000001.json

# After the import: the backfill estimate, then the same request with confirm.
curl -X POST "$API_BASE/namespaces/acme%2Fprod%2Ftenant_123/judgments/needs_escalation/backfill" \
  -H "Authorization: Bearer $VAINONA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"confirm": false}'

curl -X POST "$API_BASE/namespaces/acme%2Fprod%2Ftenant_123/judgments/needs_escalation/backfill" \
  -H "Authorization: Bearer $VAINONA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"confirm": true}'

# Follow the job it returns.
curl "$API_BASE/jobs/$JOB_ID" -H "Authorization: Bearer $VAINONA_API_KEY"
```

## Checklist

* [ ] A namespace of your own, not `default/quickstart`; one per tenant.
* [ ] Stable ids from your source, prefixed if they can collide; pointers to other documents hold their exact ids.
* [ ] Filterable fields and joins in `attributes`, what judgments read in `state`.
* [ ] Each record's original `created_at`.
* [ ] The import helper, with a few writes in flight and a raised retry count, or your own full batches of upserts with one `Idempotency-Key` each.
* [ ] The summary's `failures` fixed and imported again.
* [ ] A budget with `on_exceeded: "pause"` on the namespace before judging starts.
* [ ] For a large import: import first, then create judgments, then one backfill each, confirmed after reading its estimate.
* [ ] For relations: the `reference_index` jobs done before the backfill.
* [ ] Progress checked with `GET /namespaces/{ns}`, `GET /jobs/{id}` and the answers' freshness.