Databricks deployment

Fabric Experiments uses Cloudflare for the production Studio and hosted API, Databricks Lakebase for operational Postgres state, and Databricks SQL for production-scale warehouse aggregation. The combined Databricks App remains a rollback and owner-governed migration target during the cutover window.

Production Databricks App

The repository root is a Databricks Asset Bundle:

• databricks.yml defines targets and the production URL,
• deploy/databricks/resources/app.yml defines the App, Lakebase, SQL warehouse, ingestion volume, secret bindings, and permissions,
• deploy/databricks/resources/analytics.yml defines the Unity Catalog schema and managed volume, serverless SQL warehouse, idempotent table-initialization Job, Lakeflow telemetry pipeline, and scheduled refresh Job,
• app.yaml starts the combined Studio/API runtime,
• scripts/build-workspace.mjs builds the pnpm workspace in the Apps build container,
• scripts/start-databricks-app.mjs exchanges Lakebase OAuth credentials, verifies the migration ledger without issuing DDL, starts both Next.js services, and exposes the same-origin proxy.

The durable Harness worker is intentionally a separate, customer-owned bundle at deploy/databricks/harness-worker. See Databricks Harness worker for its secret, App-to-App OAuth, SQL warehouse, model endpoint, and Temporal requirements.

Users open studio.experiments.fabric.pro, which is served directly by Cloudflare. The Databricks-assigned App URL is an operator-only rollback endpoint.

Required keys in the fabric-experiments secret scope:

• better-auth-secret,
• manifest-key-enc-key,
• manifest-worker-hmac-secret,
• worker-api-token,
• stripe-webhook-secret and stripe-api-key (test-mode values until the production billing gate is approved),
• resend-api-key,
• resend-from,
• r2-access-key-id and r2-secret-access-key (bucket-scoped, object-read-only).

Apply migrations with a privileged installer identity before deploying the App. The installer must own existing objects or otherwise have the DDL privileges required by every pending migration. The App service principal is a runtime identity and does not perform schema changes during startup.

DATABASE_PROVIDER=databricks-lakebase \
DATABRICKS_LAKEBASE_HOST=... \
DATABRICKS_LAKEBASE_DATABASE=... \
DATABRICKS_LAKEBASE_USER=... \
DATABRICKS_LAKEBASE_ENDPOINT=projects/.../branches/.../endpoints/... \
DATABRICKS_HOST=https://... \
DATABRICKS_TOKEN=... \
FX_MIGRATION_RUNTIME_ROLE=<control-app-service-principal-uuid> \
pnpm migrate

pnpm migrate:verify

Deploy the bundle once and initialize the analytical tables:

databricks bundle deploy -t prod -p fabric-harness
databricks bundle run analytics_initialize -t prod -p fabric-harness
databricks bundle run experiments -t prod -p fabric-harness

Use the bundle resource to create the App snapshot. A direct databricks apps deploy reads app.yaml but does not apply the bundle's deployment-specific config, including the worker-dispatch mode, production App kind, catalog/schema, and current URLs.

With the direct bundle engine, databricks bundle deploy synchronizes the App resource configuration and source; databricks bundle run experiments submits the snapshot deployment. Invoke that run exactly once. If the CLI wait times out, query the deployment ID and keep polling the server-side deployment instead of running the command again. After terminal success, run an authenticated /api/healthz probe. If a deployment remains in the same preparation phase beyond the release timeout, capture its deployment ID and status before stopping and starting the App and submitting one replacement deployment.

Databricks Apps enforces workspace OAuth before requests reach the application. Use /api/healthz for an authenticated health probe; the gateway can block unauthenticated probes before the app sees them.

Customer workspaces and Marketplace

Each customer installation is an independent production cell in the customer's Databricks workspace. It uses customer-selected Lakebase, SQL warehouse, Unity Catalog, storage, secret scope, and App URL resources. It must not reuse Fabric production workspace IDs, service-principal IDs, storage accounts, or database resource paths.

Before a direct workspace installation, copy deploy/databricks/marketplace/customer-install.example.json, replace every example value, and run:

pnpm databricks:customer:preflight -- path/to/customer-install.json

The same configuration and validation contract is the source for the eventual Marketplace setup flow. Marketplace changes the delivery and consent surface, not the application boundary: the App still runs on customer compute, uses its dedicated service principal, binds customer resources, and requires privileged setup to apply migrations before runtime startup. See the customer installation runbook.

Managed analytics plane

The bundle creates:

• ${var.analytics_catalog}.${var.analytics_schema} for governed telemetry and results,
• ${var.analytics_catalog}.${var.telemetry_schema} on the customer-owned ${var.telemetry_storage_root} for Databricks Apps OpenTelemetry export,
• a managed ingestion Volume with exposures/ and conversions/ prefixes,
• a 2X-Small serverless Photon warehouse with ten-minute auto-stop,
• fabric-experiments-<target>-telemetry, a triggered Advanced Lakeflow pipeline,
• fabric-experiments-<target>-analytics-initialize, an idempotent Job that creates exposures, conversions, eval_scores, and spans before the App begins serving warehouse-backed API requests,
• fabric-experiments-<target>-analytics-refresh, an active 15-minute schedule.

On Azure Databricks, infra/databricks/r2_sync.py first copies immutable exposure and conversion batches from the Cloudflare R2 landing bucket into the governed bundle Volume. The task is idempotent and uses a bucket-scoped, read-only credential from Databricks Secrets. The downstream infra/databricks/auto_loader.py pipeline then reads the Volume and writes Delta tables. This explicit copy is required because Databricks' native Cloudflare R2 external-location integration is available on Databricks on AWS, not Azure Databricks.

telemetry_storage_root must be a customer-owned path contained by a Unity Catalog external location on which the deployer has CREATE MANAGED STORAGE. Databricks Apps rejects telemetry tables backed by workspace default storage. The dedicated schema keeps those tables managed while satisfying the custom storage boundary; override the target value for every customer workspace.

For a Databricks on AWS deployment, you can instead create a Cloudflare R2 storage credential and external location, set analytics_source_path to its r2://... location, and remove the copy task.

databricks bundle validate -t prod
databricks bundle deploy -t prod
databricks bundle run analytics_initialize -t prod

The App receives DATABRICKS_WAREHOUSE_ID and the governed Volume path through resource bindings; no warehouse token is stored in an App secret.

External warehouse aggregation

For an externally managed staging catalog/schema instead of the bundle resources:

CREATE CATALOG IF NOT EXISTS fx_staging;
CREATE SCHEMA IF NOT EXISTS fx_staging.fx;
GRANT USAGE ON CATALOG fx_staging TO `fx-staging-sp`;

Set API env vars:

• DATABRICKS_HOST
• DATABRICKS_HTTP_PATH
• DATABRICKS_TOKEN
• DATABRICKS_CATALOG=fx_staging
• DATABRICKS_SCHEMA=fx

Install the optional driver in deployments that use Databricks:

pnpm add -F @fabricorg/experiments-api @databricks/sql