Legacy Databricks service ingress

The production API now runs directly on Cloudflare. This Worker is retained without a public route as a rollback adapter for the combined Databricks App. Do not deploy its old api.experiments.fabric.pro custom domain during normal operation.

Route allowlist

GET /healthz reports only ingress-process liveness. Every other route and method returns 404, including the ordinary /v1/orgs/* API.

Security model

Databricks token-authenticated App endpoints must be below /api/. The combined App proxy maps /api/internal/* at the gateway to the existing internal API handlers. The ingress uses a dedicated Databricks service principal with only CAN USE on the production App. Do not reuse the App's own service principal, a human identity, or a workspace-admin principal.

Databricks OAuth access tokens are cached only in Worker isolate memory and refreshed before their one-hour expiry. Client credentials and shared worker tokens are Worker secrets and are never logged. Requests are capped at 1 MiB, rate-limited, and returned with no-store and MIME-sniffing protections.

One-time Databricks setup

1. Create a dedicated service principal such as fabric-experiments-service-ingress and assign it to the production workspace.
2. Generate an OAuth secret and retain the client ID and secret in the approved secret manager. The secret is displayed once.
3. Open the production App, select Share, and grant that service principal CAN USE—not CAN MANAGE.
4. Put the shared API worker token and Stripe test secrets into the Databricks secret scope used by the bundle:

databricks secrets put-secret fabric-experiments worker-api-token
databricks secrets put-secret fabric-experiments stripe-webhook-secret
databricks secrets put-secret fabric-experiments stripe-api-key
databricks bundle deploy -t prod

The bundle exposes those values to the App as WORKER_API_TOKEN, STRIPE_WEBHOOK_SECRET, and STRIPE_API_KEY through secret resource bindings.

Configure Cloudflare secrets

Use the same WORKER_API_TOKEN value in the App, ingress, audit-export worker, and OTLP worker. Do not paste secret values into wrangler.toml.

cd cloudflare/service-ingress-worker
pnpm exec wrangler secret put DATABRICKS_CLIENT_ID
pnpm exec wrangler secret put DATABRICKS_CLIENT_SECRET
pnpm exec wrangler secret put WORKER_API_TOKEN

The non-secret workspace host and App origin are committed in wrangler.toml. Confirm them against the deployed workspace and App before each production rollout.

Verify before deployment

pnpm --filter @fabricorg/cf-service-ingress-worker test
pnpm --filter @fabricorg/cf-service-ingress-worker type-check
node --test scripts/databricks-app-proxy.test.mjs

Manually obtain a token with the new principal and verify CAN USE before putting traffic on the custom domain:

curl --request POST \
  --url "$DATABRICKS_HOST/oidc/v1/token" \
  --user "$DATABRICKS_CLIENT_ID:$DATABRICKS_CLIENT_SECRET" \
  --data 'grant_type=client_credentials&scope=all-apis'

Deploy and smoke

Rollback deployment restores the api.experiments.fabric.pro custom domain. Do this only after the App has the worker and Stripe secrets, the M2M permission probe passes, and the Cloudflare API Worker has been removed from that domain.

pnpm --filter @fabricorg/cf-service-ingress-worker deploy
curl -fsS https://api.experiments.fabric.pro/healthz | jq .

Expected negative probes:

test "$(curl -sS -o /dev/null -w '%{http_code}' \
  https://api.experiments.fabric.pro/v1/orgs/acme/experiments)" = "404"

test "$(curl -sS -o /dev/null -w '%{http_code}' -X POST \
  https://api.experiments.fabric.pro/internal/otlp-ingest)" = "401"

Then send a Stripe test-mode event from the Stripe CLI or Dashboard and require 200 in Stripe's delivery log. A fabricated signature is expected to reach the App and return 400; it is not a valid positive smoke.

Stripe Customers and Subscriptions must both carry the immutable Fabric organization id in metadata.fabric_org_id. Subscription events establish the Stripe-id mapping; invoice events use their embedded subscription metadata or that existing mapping. Missing, unknown, or conflicting mappings fail closed with a non-2xx response so Stripe retries instead of writing across tenants.

Rotation and rollback

Rotate the M2M secret before its expiry by creating a second Databricks OAuth secret, updating DATABRICKS_CLIENT_SECRET, verifying the smoke, and only then revoking the old secret. Rotate WORKER_API_TOKEN across the App and all three workers in one maintenance window.

To roll back the ingress code, use wrangler rollback for fabric-experiments-service-ingress. To remove public ingress entirely, remove the custom domain route or disable the Worker; the interactive Databricks App and edge assignment/telemetry paths remain independent.