Evaluate AI applications

Fabric Experiments treats AI evaluation as another form of test evidence. It builds on managed MLflow, Unity Catalog, Model Serving, and AI Gateway rather than recreating them. A/B results tell you which experience performs better; BDD and live checks tell you whether a workload behaves correctly; AI evaluations tell you whether probabilistic output is correct, grounded, relevant, safe, and reproducible.

The AI Quality packages are TypeScript-first and provider-neutral at their core. Databricks Model Serving is available through an explicit adapter and is covered by the Databricks live-suite contract.

If the evaluation already runs in managed MLflow, keep it there and link the native run into Quality Center. Use the evaluator packages below for application-side TypeScript checks and local CI—not as a substitute for native MLflow evaluation or governance.

Install the foundations

pnpm add \
  @fabricorg/experiments-datasets \
  @fabricorg/experiments-evals \
  @fabricorg/experiments-prompts \
  @fabricorg/experiments-traces

Add the Databricks adapter only where model calls are made:

import { DatabricksModelClient } from '@fabricorg/experiments-evals/databricks';

The provider-neutral @fabricorg/experiments-evals root does not export a Databricks client. This keeps code evaluators, aggregation, and test fixtures portable.

1. Freeze a reproducible dataset

import {
  InMemoryDatasetStore,
  parseJsonl,
} from '@fabricorg/experiments-datasets';

const tenantId = 'customer-a';
const store = new InMemoryDatasetStore();
await store.create(
  tenantId,
  { id: 'support-golden', name: 'Support golden set' },
  new Date().toISOString(),
);

const imported = parseJsonl(
  '{"input":"How do I reset MFA?","expectedOutput":"Use the security settings page."}',
  () => crypto.randomUUID(),
);
if (imported.errors.length > 0) throw new Error(imported.errors[0]?.message);
await store.addExamples(
  tenantId,
  'support-golden',
  imported.examples,
  new Date().toISOString(),
);
const version = await store.createVersion(
  tenantId,
  'support-golden',
  new Date().toISOString(),
  'release candidate',
);

Dataset inputs, expected outputs, and metadata must be JSON-compatible. A version records a content hash and owns its stored values, so mutating a caller object cannot rewrite frozen evidence.

2. Choose evaluators

Seven built-in judges cover common LLM quality questions:

• hallucination;
• question-answer correctness;
• relevance;
• toxicity;
• summarization faithfulness;
• reference-grounded correctness;
• agent trajectory coherence.

Use a code evaluator for deterministic rules:

import { codeEvaluator } from '@fabricorg/experiments-evals';

const exactMatch = codeEvaluator('exact-match', '1.0.0', (item) => ({
  label: item.output === item.expectedOutput ? 'match' : 'mismatch',
  score: item.output === item.expectedOutput ? 1 : 0,
}));

Every result includes evaluator name, evaluator version, latency, and—when a model judge is used—prompt hash, serving endpoint, and token usage.

3. Run against Databricks Model Serving

import {
  executeEvalRun,
  qaCorrectnessReferenceJudge,
} from '@fabricorg/experiments-evals';
import { DatabricksModelClient } from '@fabricorg/experiments-evals/databricks';

const model = new DatabricksModelClient({
  host: process.env.DATABRICKS_HOST!,
  endpoint: process.env.DATABRICKS_SERVING_ENDPOINT!,
  auth: {
    kind: 'oauth-m2m',
    clientId: process.env.DATABRICKS_CLIENT_ID!,
    clientSecret: process.env.DATABRICKS_CLIENT_SECRET!,
  },
});

const result = await executeEvalRun({
  evaluators: [qaCorrectnessReferenceJudge, exactMatch],
  items: [
    {
      itemId: crypto.randomUUID(),
      input: {
        input: 'How do I reset MFA?',
        output: 'Open security settings and select Reset MFA.',
        expectedOutput: 'Use the security settings page.',
      },
    },
  ],
  ctx: { model, nowMs: Date.now },
  maxJudgeTokens: 10_000,
  concurrency: 4,
});

Judge requests use temperature: 0 and a provider-enforced output ceiling. Before each call, the runner reserves a conservative input-specific maximum. Concurrent calls cannot collectively cross the configured judge budget. Code and heuristic evaluators consume no judge reservation and continue even when the model budget is exhausted.

For a live certification run, configure DBX_TEST_EVAL_SERVING_ENDPOINT and run the Databricks live suite. The model-eval-judge check verifies endpoint readiness, deterministic chat payload support, label parsing, and integer token usage.

4. Preserve prompt and trace provenance

@fabricorg/experiments-prompts renders typed chat templates, discovers missing variables, and creates content hashes for version identity.

@fabricorg/experiments-traces decodes OTLP/HTTP JSON with OpenInference attributes. It retains span input, output, model, token counts, session and experiment correlation, arbitrary attributes, and span events. Unix-nanosecond timestamps remain strings so 2026-era values do not lose precision.

import { decodeOtlpJson, summarizeTraces } from '@fabricorg/experiments-traces';

const decoded = decodeOtlpJson(tenantId, otlpExportRequest);
if (decoded.errors.length > 0) console.error(decoded.errors);
const summaries = summarizeTraces(decoded.spans);

OpenInference and OTLP are used as portable evidence formats. Fabric Experiments is not intended to replace a general-purpose APM or log-management system.

5. Store evidence without implementing unrelated capabilities

The warehouse package exposes composable contracts:

• ExperimentEventStore for exposures and conversions;
• ExperimentAnalyticsStore for outcome analytics;
• EvalScoreStore for evaluator evidence and rollups;
• TraceAnalyticsStore for trace evidence and cost/error/token rollups;
• WarehouseAdapter when one implementation supplies all capabilities.

Eval scores use a tenant-scoped idempotency key. Databricks persists them with an atomic Delta MERGE; DuckDB uses a composite key; the local NDJSON adapter serializes concurrent writes within the process. Retrying a partially delivered worker batch does not duplicate evidence or allow one tenant's run identifier to suppress another tenant's result.

How AI Quality fits the product

Dataset + prompt version
        → model or agent invocation
        → OpenInference trace
        → code and model evaluators
        → score, cost, latency, and error evidence
        → Quality Center and governed release decision

The packages on this page provide the reproducible evaluation foundation. Use Databricks-native AI quality to connect managed MLflow evidence without copying its artifacts, use Databricks testing for deterministic workload behavior and Quality Center for tenant-scoped test evidence and production promotion policies.