Live artifact checks

Status: The gated runner and JSON/JUnit evidence cover SQL, Delta contracts, Unity Catalog allow/deny, Jobs, Lakeflow, Lakebase credential and pool refresh, Volumes, Apps, secret metadata, isolated Auto Loader, serverless notebook-submit, serverless dbt workloads, performance budgets, failure recovery, secret rotation, Delta backup/restore and rollback, plus explicit Azure serverless/classic compute certification. Stable checks are promoted into the required production-nightly set after their observation window.

The live tier runs against a real workspace, is always gated (silent no-op without DBX_TEST_LIVE=1), and produces evidence you can hand to an auditor.

A gated no-op exits successfully with zero live checks. That protects local development, but it is not certification evidence. Required CI workflows must set and validate DBX_TEST_LIVE=1 before accepting the run.

The check runner

import { defineLiveSuite } from '@fabricorg/databricks-testkit'

const suite = defineLiveSuite({
  required: ['sql', 'uc-grants', 'delta-contract'],
  checks: [
    restrictedPrincipalCheck(),
    sqlRoundTripCheck(),
    unityCatalogAccessCheck(),
    deltaContractCheck('DBX_TEST_DELTA_TABLE', [
      { name: 'experiment_id', type: 'string' },
    ]),
    jobRunCheck(),
    pipelineRefreshCheck(),
    lakebaseCredentialCheck(),
    lakebaseRoundTripCheck(),
    volumeIOCheck(),
    secretScopeCheck(),
    appHealthCheck(),
    autoLoaderIngestionCheck(),
    serverlessComputeCheck(),
    classicComputeCheck(),
    performanceBudgetCheck(),
    failureRecoveryCheck(),
    secretRotationCheck(),
    backupRestoreCheck(),
    rollbackCheck(),
  ],
})

Each check reports pass | fail | not-configured with durations; required checks gate the exit code, optional ones inform. Output is a schema-versioned, secret-redacted evidence JSON plus JUnit XML for CI annotation.

For productized workload groups, prefer a target pack. Packs reuse this runner and add version, maturity, capabilities, requirements, certification scope, and target metadata without breaking existing evidence readers:

import { builtinTargetPacks, runTargetPack } from '@fabricorg/databricks-testkit'

const sqlDelta = builtinTargetPacks.find((pack) => pack.id === 'sql-delta')!
await runTargetPack(sqlDelta, {
  evidencePath: 'reports/sql-delta.json',
  junitPath: 'reports/sql-delta.xml',
})

Use fx targets list, fx targets doctor <pack>, and fx targets run <pack> --live for the supported CLI workflow.

Built-in checks

The bundle includes a separate ${analytics_schema}_bdd schema, Volume, and pipeline. Auto Loader full refreshes are rejected when the configured pipeline matches the production pipeline. Product-specific checks can be registered with customLiveCheck() and receive the shared evidence and required-check semantics.

Long-running checks use poll-to-terminal helpers with explicit timeouts and abort support — a hung pipeline fails the check rather than the CI job's global timeout.

Environment

DBX_TEST_LIVE=1
DATABRICKS_HOST=https://<workspace>.cloud.databricks.com
DATABRICKS_HTTP_PATH=/sql/1.0/warehouses/<id>   # optional; discovered if unset
# auth: one of
DATABRICKS_BEARER=...        # CI (pre-minted M2M token)
DATABRICKS_CLIENT_ID=... DATABRICKS_CLIENT_SECRET=...
DATABRICKS_TOKEN=...         # PAT (local development)
DBX_TEST_CATALOG=fx_test     # scratch catalog for scenario schemas
DBX_TEST_DELTA_TABLE=<catalog>.<schema>.<table> # required by delta-contract
DBX_TEST_PIPELINE_ID=...     # optional; required when `pipeline` is a gate
DBX_TEST_LAKEBASE_ENDPOINT=projects/.../branches/.../endpoints/...
DBX_TEST_LAKEBASE_PROJECT=projects/<project> # cloned by `fx test env` per run
DBX_TEST_LAKEBASE_HOST=... DBX_TEST_LAKEBASE_DATABASE=... DBX_TEST_LAKEBASE_USER=...
DBX_TEST_VOLUME=/Volumes/<catalog>/<bdd-schema>/ingestion
DBX_TEST_AUTOLOADER_PIPELINE_ID=... DBX_TEST_AUTOLOADER_TABLE=<catalog>.<bdd-schema>.exposures
DBX_TEST_SECRET_SCOPE=fabric-experiments DBX_TEST_SECRET_KEYS=key-a,key-b
DBX_TEST_APP_NAME=fabric-experiments-prod DBX_TEST_APP_RESOURCES=binding-a,binding-b
DBX_TEST_RESTRICTED_PRINCIPAL_ID=<workload-service-principal-application-id>
DBX_TEST_NOTEBOOK_PATH=/Workspace/Shared/... DBX_TEST_NOTEBOOK_SERVERLESS=1
DBX_TEST_DBT_PROJECT_DIR=/Workspace/Shared/... DBX_TEST_DBT_SERVERLESS=1
DBX_TEST_PERFORMANCE_P95_MS=15000 DBX_TEST_PERFORMANCE_RUNS=7
DBX_TEST_ROTATION_SECRET_SCOPE=fabric-experiments-nightly
DBX_TEST_CLASSIC_SPARK_VERSION=16.4.x-scala2.12
DBX_TEST_CLASSIC_NODE_TYPE=Standard_D2ds_v6 DBX_TEST_CLASSIC_WORKERS=0
DBX_TEST_CLASSIC_SINGLE_USER_NAME=<service-principal-application-id>
# Optional comma-separated capacity fallbacks for multi-node classic clusters.
DBX_TEST_CLASSIC_ALTERNATE_NODE_TYPES=Standard_DS3_v2

CI wiring

A nightly workflow (databricks-live.yml) uses a bootstrap OAuth identity to create an isolated UC schema and cloned Lakebase branch, then switches to a dedicated least-privilege OAuth identity for the live suite and product BDD features. It uploads JSON/JUnit/HTML evidence, purges both scratch resources in teardown, and alerts Slack on failure. See Live smoke and promotion for the release gate.

Cost controls: serverless SQL warehouse with auto-stop, scenario-scoped scratch schemas dropped in teardown, and @slow checks confined to the nightly run.

The separate databricks-classic.yml workflow runs the REST-only classic check without requiring a SQL warehouse and publishes JSON/JUnit evidence weekly. For Azure VNet-injected workspaces with no public node IPs, provide explicit outbound NAT and keep requiredNsgRules=AllRules unless a complete Private Link design replaces public control-plane connectivity. The certified legacy SCC workspace also permits the relay port reported by its Databricks bootstrap diagnostics.

Browser-to-Studio data-path certification

The staging gate automates the complete managed path:

node scripts/live-smoke-browser-databricks-studio.mjs

Headless Chrome emits an exposure with explicit segment dimensions and a conversion to the staging Cloudflare worker. The script starts the dedicated browser-to-studio-e2e Databricks job, which syncs the staging R2 bucket into the isolated BDD Volume and refreshes Auto Loader. It then queries Delta for the unique browser subject, calls the aggregate API, signs into Studio, and requires the Bayesian, CUPED, time/depth, and segment result surfaces to render. Any missing credential or target fails configuration validation; no stage silently skips.

The older live-smoke-production-edge.mjs remains the fast CDN/tag/anti-flicker probe. It is not used as evidence for the complete warehouse path.