Cloudflare workers

Edge workers serve manifests, ingest exposure data, export audit events, and host this docs site.

Fabric Experiments already includes Cloudflare worker packages.

Manifest worker

cloudflare/manifest-worker serves signed manifests from KV:

GET /manifest
GET /{tenantId}/manifest
GET /{tenantId}/manifest/v/{version}
POST /admin/publish
POST /admin/tenants/register (provider administration)
DELETE /admin/tenants/{tenantId} (provider administration)
GET /.well-known/jwks/{orgId}
GET /.well-known/revocations/{orgId}

Publishing is HMAC-protected in staging/production. Registered customer tenants use an independent tenant HMAC; the provider HMAC is accepted only for provider administration and legacy unregistered tenants.

Use tenant-scoped /{tenantId}/manifest URLs in every multi-tenant deployment. The legacy /manifest alias is enabled only when DEFAULT_TENANT is explicitly set; without it, the worker fails closed. Never set this variable on a shared worker unless every unscoped caller is intended to read that one organization, because a wrong value cross-routes delivery.

Exposure worker

cloudflare/exposure-worker accepts:

POST /{tenantId}/v1/exposure
POST /{tenantId}/v1/conversion
GET /{tenantId}/v1/delivery?kind=exposures|conversions
GET /{tenantId}/v1/delivery/object?key=...
DELETE /{tenantId}/v1/delivery/object?key=...

The URL tenant is authoritative. The worker enforces tenant-scoped rate limits, request and batch limits, validates events, and acknowledges with 204 only after the batch is durably accepted by Cloudflare Queues. The queue consumer writes deterministic NDJSON objects to R2 and retries transient failures; exhausted messages move to fabric-experiments-telemetry-dlq for investigation.

Delivery routes are server-to-server, have no wildcard CORS, and require the registered tenant's bearer token. They list and retrieve only that tenant's R2 prefix. The customer Job deletes an object only after customer Delta materialization succeeds; interrupted runs leave it available for retry.

The legacy unscoped /v1/exposure and /v1/conversion routes follow the same rule: they exist only with an explicit DEFAULT_TENANT. Prefer the scoped routes and leave the fallback unset on shared infrastructure.

Production endpoints:

https://manifest.experiments.fabric.pro
https://ingest.experiments.fabric.pro
https://cdn.fabric.pro

Production API and legacy service ingress

The full OpenNext API Worker serves api.experiments.fabric.pro, including the Stripe webhook, audit export/forwarding, OTLP hot-store delivery, and ordinary tenant APIs. cloudflare/service-ingress-worker is retained without a route only for rollback to the combined Databricks App.

The worker exchanges a dedicated Databricks service principal's client credentials for a one-hour OAuth token and refreshes it before expiry. Internal callers still present WORKER_API_TOKEN; the ingress validates that token and moves it to x-fabric-worker-token because Databricks Apps consumes the Authorization header for gateway OAuth. The API validates the forwarded token again. Stripe's raw body and Stripe-Signature header pass through unchanged so the API remains the signature-verification boundary.

See the legacy service ingress runbook for the exact rollback routes, permissions, secrets, and deployment sequence.

OTLP worker

cloudflare/otlp-worker ingests OTLP/HTTP trace exports for AI observability:

POST /{tenantId}/v1/traces with Content-Type: application/json or application/x-protobuf (ExportTraceServiceRequest)

The URL tenant is authoritative. Protobuf payloads are transcoded at the edge by a zero-dependency wire-format reader into the OTLP JSON mapping, so both transports share one decode and validation path; malformed protobuf is rejected with 400. Accepted spans are queued; the queue consumer writes NDJSON span rows to R2 under the spans/ prefix (the lakehouse analytics copy) and then delivers the same batch to the API's internal /internal/otlp-ingest route (worker-token authenticated) so spans reach the Postgres hot store that serves Studio's trace views. Hot-store delivery failures retry through the queue; both destinations are idempotent under redelivery.

Ingest tokens

Every trace POST must carry an org-scoped, WRITE-ONLY ingest token:

Authorization: Bearer fxit_<tokenId>_<secret>

Tokens are minted and revoked in Studio → Settings → Ingest tokens (admin role). Only a SHA-256 hash is stored; the secret is shown once. On mint and revoke the API pushes the token's org binding and hash to the worker's KV over the HMAC-signed /admin/ingest-tokens endpoint (the same push pattern as manifest publishes), so edge verification needs no database round trip. The worker compares hashes in constant time and requires the token's organization to match the URL tenant: requests without a valid token get 401, tokens bound to a different organization get 403.

CORS

There is no wildcard CORS on the traces route. Cross-origin browser ingestion must be enabled per tenant with the FX_ALLOWED_ORIGINS wrangler var — a JSON map of tenant ID to allowed origins; only configured origins are reflected, and the Authorization header is allowed so browser tracers can present ingest tokens. Server-side ingestion (sdk-node, OTLP collectors) sends no Origin header and is unaffected. The browser SDK uses keepalive fetch instead of sendBeacon when a token is configured, because sendBeacon cannot set headers.

Tenant PII redaction

Per-tenant redaction policies (governance control traces.pii-handling) are configured with the FX_REDACTION_POLICIES wrangler var — a JSON map of tenant ID to policy. The default is "{}": no redaction.

{
  "acme": {
    "dropInputOutput": true,
    "attributeAllowlist": ["http.method"]
  }
}

dropInputOutput nulls the lifted input.value / output.value fields, removes them from the attribute bag, and marks spans with fx.redacted: true.
attributeAllowlist strips every non-essential attribute that is not allowlisted. Essential attributes always survive: openinference.span.kind, llm.token_count.*, llm.model_name, session.id, and fx.*.

Redaction runs after decoding and before the batch is queued, so redacted values never reach the queue, R2, or the warehouse.

Audit export worker

cloudflare/audit-export-worker runs every five minutes, polls the API for active export targets, writes destination batches, and acknowledges cursor progress.

Docs worker

cloudflare/docs-worker serves the static Fumadocs export at your documentation domain.

On this page