AI evaluations

Judge LLM outputs with built-in evaluators, durable eval runs, hard token budgets, and Quality Center evidence.

Fabric Experiments evaluates AI outputs the same way it evaluates conversion metrics: as governed, evidence-producing runs. An eval run scores a set of items with one or more evaluators and lands its verdict in the same Quality Center that gates production promotion.

Managed MLflow is the preferred system of record when the evaluation is already native to Databricks. Fabric can link a managed MLflow run into the same release gate without copying its artifacts, traces, datasets, or lineage. The Fabric evaluator runner below is the TypeScript integration path for application-side checks and local CI.

Built-in evaluators

Seven LLM judges ship out of the box, each versioned and provenance-complete (every score records the evaluator name and version, the judge prompt hash, and the model endpoint that produced it):

Evaluator	Judges	Score
`hallucination`	Is the answer grounded in the provided context?	binary
`qa-correctness`	Is the answer correct for the question?	binary
`relevance`	Is the response on-topic for the input?	binary
`toxicity`	Is the text safe?	binary
`summarization`	Does the summary faithfully cover the source?	binary
`qa-correctness-reference`	Does the answer match a reference answer?	binary; unscored when no reference is provided
`trajectory-coherence`	Does a multi-step agent transcript cohere?	binary

Judges execute against Databricks Model Serving in production (workspace-native auth — PAT or M2M OAuth); local development falls back to a deterministic static client so tests and fx dev never need a workspace.

Creating a run

In Studio: Evals → New eval run — pick evaluators, then a frozen dataset version as the target, set an optional token budget, and launch. The run page live-renders status, per-evaluator pass rates with Wilson confidence intervals, and per-item judge explanations.

Via the API:

const run = await api.evals.createRun({
  target: { kind: 'dataset', datasetId: 'support-answers', version: 3 },
  evaluatorNames: ['qa-correctness-reference', 'toxicity'],
  maxJudgeTokens: 200_000,
});
// run.status === 'pending' — execution is durable and asynchronous

Durable execution

Runs never execute inside the API request. createRun records the run and an outbox row in one transaction; a dispatcher (in-process in dev, the Temporal-backed Harness worker in production) executes the run with retries, cancellation, and idempotent redelivery. Cancel any time from the run page — partial scores are kept.

Token budgets are hard limits

maxJudgeTokens is enforced by reservation accounting: a judge call only starts if the already-spent plus reserved tokens still fit the budget, so concurrent judging can never overshoot it. Items skipped for budget are reported on the run. Budgets are ceilings, not targets — size them from your dataset size × expected per-item tokens.

Cost

Each run records cost_usd, computed from a versioned model-price manifest over the actual token usage. Cost appears on the run page and in run listings.

Hosted evaluator catalog

The hosted Fabric runner exposes the seven built-ins. Use native managed MLflow for custom scorer registration and production trace monitoring, or use the public @fabricorg/experiments-evals contract for TypeScript code evaluators in application-owned workflows.
Experiment-target runs (judging live traffic per variant) require trace ingestion to be enabled for your organization.