Datasets

A dataset is a named collection of evaluation examples. Examples are append-only; a version freezes the current membership into an immutable, content-addressed snapshot — the same discipline as the signed edge manifest, so two versions with identical membership are provably identical.

Examples

Each example carries:

• input — the prompt, question, or transcript under evaluation (any JSON)
• expectedOutput — optional reference answer, used by reference-based judges such as qa-correctness-reference
• metadata — provenance and dimensions (free-form)
• split — train, test (default), or validation

Workflow

1. Create a dataset in Studio (Datasets → New dataset) or via api.datasets.create.
2. Add examples — paste JSONL in Studio, or api.datasets.addExamples.
3. Freeze a version when the set is ready. Eval runs always target a frozen version, so results are reproducible: the same version + the same evaluator versions ⇒ the same items judged.

await api.datasets.create({ id: 'support-answers', name: 'Support answers' });
await api.datasets.addExamples('support-answers', {
  examples: [
    {
      exampleId: crypto.randomUUID(),
      input: 'What is SRM?',
      expectedOutput: 'Sample ratio mismatch.',
      metadata: {},
      split: 'test',
    },
  ],
});
await api.datasets.createVersion('support-answers', {});

Regression evals

The intended loop: freeze a version, run an eval, ship changes, re-run the same evaluators against the same version, and compare runs side by side in Studio (Evals → compare). Because versions are immutable, a score change is attributable to your system — not to dataset drift.