Live smoke gates

Live smokes are the operational gate for staging readiness.

Required GitHub Actions secrets:

• FX_LIVE_API_BASE_URL
• FX_LIVE_API_KEY
• FX_LIVE_ORG_ID
• FX_LIVE_EXPERIMENT_ID for M4
• FX_LIVE_WORKER_TOKEN for M5
• FX_LIVE_STUDIO_BASE_URL, FX_LIVE_USER_EMAIL, FX_LIVE_USER_PASSWORD
• Databricks OAuth, warehouse, and DBX_TEST_BROWSER_E2E_JOB_ID
• optional FX_LIVE_EXPECT_BREACH

Use a dedicated smoke service principal. It needs CAN MANAGE RUN on the browser ingestion job, CAN USE on the SQL warehouse, USE CATALOG on the telemetry catalog, and USE SCHEMA plus SELECT on the isolated telemetry tables that the smoke verifies. It does not need workspace-admin or catalog ownership. The ingestion job should run as its own workload identity with write access; the smoke identity only starts the job and verifies its output.

The Cloudflare API and Databricks App each use a separate workload identity for aggregate queries. Grant each identity USE CATALOG, USE SCHEMA, and SELECT on its telemetry schema. The bundle records the schema, warehouse, and job permissions; the catalog-level USE CATALOG grant remains a one-time platform bootstrap because the bundle consumes, rather than owns, the shared catalog.

The workflow .github/workflows/live-smoke.yml runs nightly and validates all configuration before any gate starts. Missing M4, M5, browser, Studio, or Databricks configuration fails the workflow; no required stage silently skips.

The Cloudflare API uses Hyperdrive with SQL response caching disabled. Mutable auth and organization reads must never be cached. Its pg pool also sets PG_POOL_MAX_USES=1, which destroys a TCP client on release so a request does not inherit an I/O handle owned by a completed Workers isolate.

If /healthz?verbose=1 can read Postgres but the first authenticated smoke call returns 500, verify the database and transaction settings:

SELECT current_setting('default_transaction_read_only'),
       current_setting('transaction_read_only');

Both values must be off for staging. If a database default was accidentally enabled, connect with an administrative session, set the session default to off, restore the database default, and recycle existing Hyperdrive origin sessions so they reconnect with the corrected setting. Do not make request handlers perform DDL as a workaround; deploys apply the canonical forward migrations before the smoke is rerun.

Every successful scheduled run writes staging-certification.json, queries the workflow history, and reports the current consecutive-green count. Production promotion remains blocked until the counter reaches seven. A failure resets the count; historical green runs are not relabeled. Manual dispatches upload path evidence but are excluded from the nightly count by checking the GitHub event.

Better Auth cookies are scoped to .experiments.fabric.pro so the API and server-rendered Studio can share a session. Staging uses the separate fx_staging_ cookie prefix, preventing production/staging session collisions. The API's CORS middleware permits credentialed requests only from configured Studio origins and returns 403 for unknown preflights.