Behavior-driven testing

Status: Available (Phase 2 shipped). Engine: cucumber-js with the Databricks step library from @fabricorg/databricks-bdd; running examples live in packages/databricks-bdd/features/ and product scenarios in apps/api/features/ (both run in CI on the local profile). Steps marked (Phase 3) below need a live workspace and ship with the live tier. Feature files are kept engine-neutral so a pytest-bdd step library can execute the same features for Python-first teams later.

BDD is the natural register for Databricks workloads: data producers, analysts, and platform engineers can all read — and agree on — a Gherkin scenario, and the same scenario is enforced against a local engine on every PR and against a real workspace nightly.

A complete example

Feature: Experiment aggregation attributes conversions correctly

  Background:
    Given catalog "fx_test" and schema "scenarios"
    And a table "exposures" with:
      | experiment_id | subject_id | variant_key | at                   |
      | checkout-cta  | u1         | control     | 2026-07-01T10:00:00Z |
      | checkout-cta  | u1         | treatment   | 2026-07-01T11:00:00Z |
      | checkout-cta  | u2         | treatment   | 2026-07-01T10:05:00Z |
    And a table "conversions" with:
      | subject_id | event_name | value | at                   | tenant_id |
      | u1         | purchase   | 49.0  | 2026-07-01T12:00:00Z | default   |

  Scenario: Conversions attribute to the first exposure
    When I run the aggregate for experiment "checkout-cta" with metric "purchase"
    Then variant "control" has 1 conversions
    And variant "treatment" has 0 conversions
    And the SRM guardrail does not fire

This exact feature runs in this repo's CI (apps/api/features/). Run it locally (DuckDB, sub-second) or against a workspace:

DBX_TEST_PROFILE=local pnpm run test:features
DBX_TEST_PROFILE=live DBX_TEST_LIVE=1 pnpm run test:features

Long-running artifacts

Live-only scenarios cover Jobs and DLT pipelines with poll-to-terminal steps:

@live @dlt
Scenario: Auto Loader lands exposure files with no schema drift
  When I start a full refresh of pipeline "fx-exposure-autoloader"
  Then the pipeline update completes within 15 minutes
  And table "exposures" contains rows matching:
    | experiment_id | subject_id |
    | checkout-cta  | u1         |
  And no rows were rescued in table "exposures"

@live @jobs
Scenario: Nightly readout job succeeds
  When I run job "fx-nightly-readout"
  Then the job run reaches SUCCESS

Step library reference

Given — environment and data setup:

• a Databricks workspace — validates auth + warehouse eagerly on the live profile; a no-op locally.
• catalog {string} and schema {string} — scopes the scenario's execution context (must precede the first warehouse-touching step).
• a table {string} with: — data table; column types inferred from cells (BIGINT/DOUBLE/TIMESTAMP/BOOLEAN/STRING, empty cells → NULL).
• a table {string} with schema {string} and rows: — explicit DDL.
• an empty table {string} with schema {string}.
• a table {string} with schema {string} from fixture {string} — NDJSON file.
• fixture files uploaded to volume {string} (Phase 3).
• service principal {string} with grants: (Phase 3).

When — actions:

• I run the SQL: (docstring; failures are captured for the the statement fails … assertions) · I query table {string}
• I run job {string} (@jobs) · I submit notebook {string} (Phase 3)
• I start a full refresh of pipeline {string} (@dlt)
• I run dbt build for {string} (Phase 3)

Then — assertions:

• table {string} has {int} rows
• table {string} contains exactly: / contains rows matching:
• table {string} matches golden {string} ordered by {string}
• the result has {int} rows / the result contains rows matching: / the result matches golden {string}
• the statement fails with {string} / the statement fails with permission denied
• the job run reaches {word} · the pipeline update completes within {int} minutes
• no rows were rescued in table {string}
• row count of {string} is within {int}% of {int}
• table {string} matches schema contract {string} (Phase 3)

Product-domain steps (aggregates, variant {string} has {int} conversions, the SRM guardrail fires / does not fire) are an extension example living in apps/api/features/support/product-steps.ts — layer your own domain steps on DatabricksWorld the same way.

Tags and profiles

Reports

Runs emit a schema-versioned, secret-redacted evidence JSON via the bundled formatter (@fabricorg/databricks-bdd/evidence-formatter, same spirit as the live check runner's output) so nightly BDD runs double as audit artifacts; add cucumber's built-in html:reports/cucumber.html format entry for the human-readable report.

Governance scenarios

Least-privilege as an executable spec (the grants step ships with the Phase 3 live tier; the statement fails with permission denied is available now):

@live
Scenario: CI principal cannot read outside its schema
  Given service principal "fx-ci" with grants:
    | securable          | privilege |
    | fx_test.scenarios  | USE, SELECT |
  When I run the SQL:
    """
    SELECT * FROM fx_prod.raw.exposures LIMIT 1
    """
  Then the statement fails with permission denied