Databricks customer installation

Fabric Experiments treats every customer workspace as an independent production cell. Application compute, operational state, analytical data, secrets, and service identities belong to that workspace. A direct workspace installation and a Marketplace installation use the same resource and migration contract; Marketplace adds Databricks packaging, consent, update notification, and security review.

Identity boundary

Use three distinct identities:

• The installer identity creates or binds resources, applies migrations, grants runtime access, and records release evidence. Before creating the delivery Job it must also hold Service Principal User (Use) on the worker App principal; Service Principal Manager (Manage) does not imply Use. It is short-lived and is not used for application requests.
• The control App service principal is minted for that installation and receives only the resources needed to serve the product. Startup executes pnpm migrate:verify, which reads fx._migrations and fails closed if the installer did not apply the required release.
• The worker App service principal is independently minted and receives only the model endpoint, warehouse, and worker secrets needed for durable evaluation dispatch and tenant telemetry delivery. The scheduled delivery Job runs as this same principal; no provider identity enters the workspace.

Do not grant the runtime principal ownership merely to make migrations work. Many PostgreSQL ALTER operations require ownership and cannot be made safe by adding broad runtime grants.

The worker is required for evaluation dispatch; disabling it is not a supported customer configuration.

Build the customer release

The checked-in app.yaml and root bundle describe Fabric's own production cell. Never distribute them as a customer artifact. Start from the customer contract, use an immutable commit, and generate the minimized release:

pnpm databricks:customer:preflight -- path/to/customer-install.json
pnpm databricks:customer:release -- \
  path/to/customer-install.json \
  dist/customer-releases/customer
pnpm databricks:customer:certify-artifact -- \
  dist/customer-releases/customer

The builder refuses a dirty repository or a release.commitSha that differs from HEAD. Its output contains minimized control/worker source archives with customer-rendered app.yaml files, the setup notebook, all migrations and checksums, required resource bindings, egress disclosures, availability policy, and a release manifest with SHA-256 checksums. It excludes Fabric workspace identifiers, environment files, node_modules, and provider deployment configuration.

pnpm databricks:customer:certify-local builds and inspects a disposable artifact using the example contract. This is deterministic offline coverage; it does not replace a live workspace lifecycle rehearsal.

An internal rehearsal can pass --internal-certification to the preflight, release, and certification commands when its Databricks URL or storage path necessarily contains Fabric workspace identifiers. The generated manifest is then marked certificationOnly: true and ordinary artifact certification rejects it. Such an artifact is evidence for the rehearsal, never a customer or Marketplace deliverable.

Direct workspace installation

1. Create the empty control and worker Apps first. This mints their dedicated service principals and gives the control App its workspace-specific URL; do not deploy source yet. For CLI automation, combine apps create --no-compute --no-wait: --no-compute can leave a successfully created App in STOPPED, which must not be treated as a creation failure.
2. Create a customer configuration from deploy/databricks/marketplace/customer-install.example.json, using those App names and the discovered control App URL. Do not put secret values in this file. Build and certify the release as described above.
3. Review resources.json, egress.json, availability.json, and release-manifest.json with the customer administrator.
4. Import and run setup_notebook.py with the two App principal IDs. The notebook creates only the declared Unity Catalog schemas, ingestion Volume, and versioned exposures, conversions, eval_scores, and spans Delta tables. It does not create secrets, deploy Apps, or apply Lakebase migrations. The Apps receive USE SCHEMA, SELECT, and MODIFY, not schema DDL.
5. Put the control secret values in the customer secret scope and bind the control resources listed in resources.json with the declared permission. Transactional email is required for verified first-user onboarding in contract version 1, so the customer must supply the declared Resend keys and approve https://api.resend.com as required egress.
6. With the privileged installer database identity, apply the artifact's ordered migrations and verify migration-manifest.json. Do not use either App principal for DDL. For Lakebase, use the direct endpoint host and the installer's complete workspace user name. Do not use the pooled host: it can reject the migration runner's startup search_path option before SQL runs. Set FX_MIGRATION_RUNTIME_ROLE to the control App service-principal UUID. The installer then grants only schema usage, DML on the fx tables and the explicit public.fx_api_keys compatibility table, plus sequence access; it does not grant schema creation, ownership, or DDL to the App.
7. Extract and deploy control-plane-source.tar. Complete first-user verification and create the customer organization. Generate independent manifest HMAC and telemetry delivery secrets and complete provider tenant registration. The provider stores only the delivery-token digest and never exposes its global administrative credential to the workspace.
8. Create a write-scoped worker API key, and store the organization ID and API key as the worker-organization-ids and harness-fx-api-key secrets. Tenant identity is discovered after onboarding and is never baked into the release.
9. Bind every worker resource listed in resources.json, including the worker's CAN USE binding on the control App, then extract and deploy worker-source.tar. Submit each deployment exactly once and retain its immutable deployment ID if the CLI wait times out. After apps start, list server-side deployments before submitting: Databricks can automatically redeploy the last source snapshot as part of the start operation.
10. Explicitly grant the installer Service Principal User (Use) on the worker App service principal while preserving any existing grants. Upload customer_delivery_sync.py, replace the explicit placeholders in delivery-job.template.json, and create the scheduled serverless Job. It runs as the worker App principal and reads the delivery token from the customer secret scope. The delivery stream writes the installer-created Delta tables and does not require schema-creation permission.
11. Poll both Apps to terminal success. Verify authenticated /api/healthz, sign-in, one governed write, a browser exposure delivered into the customer-owned Delta table, one warehouse-backed aggregate, and an evaluation completed by the worker.

Required public delivery boundary

The v1 customer contract supports delivery.mode=fabric-managed. Databricks Apps ingress is workspace-authenticated, while browser manifest retrieval and exposure beacons need public endpoints. The release therefore requires:

• cdn.fabric.pro for immutable web SDK assets,
• manifest.experiments.fabric.pro for signed public manifests,
• ingest.experiments.fabric.pro for browser exposure ingestion, and
• the customer-selected Temporal endpoint for durable worker orchestration.

egress.json states the purpose and data categories for each origin. Marketplace or direct-install consent must disclose that published experiment configuration and exposure evidence cross the workspace boundary. A workspace-only public delivery mode is not currently certified and fails preflight instead of silently falling back to Fabric endpoints.

The public edge is a durable handoff, not the system of record. The customer delivery Job lists and downloads only the registered tenant prefix, validates every row's tenant identity, writes it to the governed ingestion Volume, and uses available-now Auto Loader checkpoints to materialize Delta. Only after both exposure and conversion streams succeed does it acknowledge the provider objects. A retry therefore remains at-least-once at the handoff and idempotent at the customer table. Certification must also prove that another tenant's token and object key are rejected.

Marketplace installation

The Marketplace package must express the same resources and scopes as the direct installation. Its setup workflow has separately auditable privileged phases: the supplied notebook creates the customer-owned Unity Catalog resources, and an installer-owned database phase applies Lakebase migrations. The App starts only after both phases succeed and migration verification passes.

Customers can choose whether optional billing egress is enabled. Transactional email is required in contract version 1 because first-user onboarding requires email verification; Databricks-native SSO is not yet an alternative identity bootstrap. Contract version 1 rejects provider telemetry because it does not yet define a destination and data-use contract. The public delivery origins above remain required for the advertised browser experimentation workflow. Required and optional network origins are disclosed separately so administrators with restricted egress can make an informed allowlist decision.

Third-party Marketplace Apps execute on customer compute in the customer's workspace. Databricks performs a security review before publication, and customers configure resources and scopes during installation. See the official Marketplace installation guide and Databricks App resource guidance. The exact provider upload and listing workflow remains gated by Databricks provider onboarding; do not invent a Marketplace package format that is not in the partner specification.

Upgrade

Use expand-and-contract migrations:

1. Apply backward-compatible expansion migrations with the installer identity.
2. Verify the ledger.
3. Deploy exactly one new App version and complete health and behavior checks.
4. Remove obsolete columns or constraints only in a later release after the supported rollback window closes.

The verifier accepts ledger entries newer than the current binary, which allows an N-1 binary to run during a rollback. It rejects any migration required by the current binary that is absent.

Failure and recovery

Use a bounded deployment timeout. If status and update time remain unchanged through that window:

1. Record the App name, deployment ID, release commit, phase, timestamps, and sanitized status output.
2. Confirm no other deployment is active.
3. Stop and start App compute.
4. Submit one replacement deployment and poll it to a terminal state.

Never solve a build or deployment stall by submitting concurrent deployments. The previous healthy artifact remains the rollback target, but it is not still serving after an in-place update has stopped App compute.

Update availability

The certified availability policy is maintenance-window, not zero downtime. The 2026-07-30 internal deployment stopped the active App before source download, build, migration verification, and startup completed. Until a customer-workspace blue/green design is proven, notify users and schedule updates inside a maintenance window. Marketplace update notification does not remove this requirement.

Uninstall

Removing the App must not delete customer Lakebase data, Unity Catalog data, or storage by default. Produce an inventory and explicit retention/deletion choices. Databricks Lakebase resources persist across App redeploys and stops; database-role ownership may need explicit reassignment when a resource is removed.