Ephemeral test environments

Status: Available. Unity Catalog scratch schemas, Lakebase branch clones, JSON fixture seeding, TTL metadata/reaping, exact teardown, and the reusable workflow are implemented. The production nightly uses all of them.

The production bundle also provisions a long-lived, isolated BDD schema, ingestion Volume, and Lakeflow pipeline. Nightly Auto Loader features use those resources and a per-run directory prefix; they never upload fixtures to or full-refresh the production telemetry pipeline.

Shared, long-lived test workspaces rot: fixtures drift, grants accumulate, and tests start depending on state nobody remembers creating. The ephemeral tier gives every pull request its own scratch environment.

CLI

fx test env create --name pr-1234 --seed fixtures/ --ttl 24h \
  --principal <service-principal-application-id> \
  --skip-catalog-grant \
  --lakebase-branch --lakebase-project projects/fabric-experiments
fx test env list
fx test env destroy --name pr-1234

create provisions:

• a Unity Catalog schema fx_pr_1234 (optionally its own catalog), with USE/SELECT/MODIFY granted only to the CI service principal;
• optionally a TTL-scoped Lakebase branch for control-plane database tests;
• seeded fixture tables from --seed;
• an environment manifest recording everything created, so destroy is exact.

The branch endpoint, host, and database are returned in the environment JSON, so CI can point the production Lakebase provider at the clone without touching the production branch. Environments carry a TTL; explicit teardown deletes the schema and purges the branch, while Lakebase TTL is a second cleanup boundary for interrupted jobs.

Reusable CI workflow

Any repository can consume the published workflow:

jobs:
  databricks-tests:
    uses: Fabric-Pro/fabric-experiments/.github/workflows/databricks-test.yml@main
    with:
      seed: fixtures/
      catalog: tf_databricks
      warehouse_id: 599af4d9378acba2
      lakebase_project: projects/fabric-experiments
      principal: <service-principal-application-id>
      test_command: pnpm test:features
    secrets:
      DATABRICKS_HOST: ${{ secrets.DATABRICKS_HOST }}
      DATABRICKS_CLIENT_ID: ${{ secrets.DATABRICKS_CLIENT_ID }}
      DATABRICKS_CLIENT_SECRET: ${{ secrets.DATABRICKS_CLIENT_SECRET }}

The workflow runs, in order:

1. local-profile BDD features and unit suites (no workspace needed — always on);
2. fx test env create for the PR;
3. live-profile BDD features and the live check suite inside that environment;
4. fx test env destroy, unconditionally (if: always()).

Scoping rules

• One environment per PR; scenario-scoped scratch schemas nest inside it.
• The workload principal's grants are confined to the PR schema — enforced by the governance scenarios in the BDD suite, so a grant misconfiguration is itself a test failure.
• Use --skip-catalog-grant when a least-privilege bootstrap identity creates schemas but cannot manage the parent catalog. Provision the workload identity's USE CATALOG once, then let each run grant only its new schema.
• Production nightlies use separate bootstrap and workload credentials. Setup and teardown use the bootstrap identity; every assertion uses a dedicated service principal, and restrictedPrincipalCheck() fails closed if that switch did not occur.
• Fixtures are committed files; an environment is fully reproducible from the repo at any commit.