FabricFabricExperiments
Deployment

Temporal and Harness worker

Deploy the durable SRM watcher, analyst, and eval worker.

The production worker runs with FX_HARNESS_RUNTIME=temporal and connects to Temporal Cloud. One process polls all three task queues by default:

  • srm-watcher-cron — a five-minute schedule per configured organization
  • analyst-signal — concluded and killed experiment readouts
  • eval-run — durable evaluation runs from the control-plane outbox

Every Temporal activity invokes the corresponding Fabric Harness 4.2 DefinedAgent; the durable path does not bypass the agent definition. With FX_AGENT_CONTROL_PLANE=1, the worker also bootstraps the certified organization registrations and receives fail-closed admission for the pinned implementation digest before each activity executes.

Required env vars:

  • FX_HARNESS_RUNTIME=temporal
  • TEMPORAL_ADDRESS
  • TEMPORAL_NAMESPACE
  • TEMPORAL_CLOUD_API_KEY, or TEMPORAL_CLIENT_CERT and TEMPORAL_CLIENT_KEY
  • FX_API_BASE_URL
  • FX_API_KEY
  • WORKER_API_TOKEN
  • FX_ORG_IDS with a comma-separated organization list
  • FX_AGENT_CONTROL_PLANE=1 for production admission and run attestations
  • optional FX_HARNESS_WORKER_BUILD_ID identifying the immutable release
  • FX_DATABRICKS_APP_AUTH=1 and FX_DATABRICKS_APP_BASE_URL when the API is a separate Databricks App
  • optional FX_HARNESS_TASK_QUEUE for one queue
  • optional FX_HARNESS_TASK_QUEUES for a comma-separated queue subset
  • optional FX_AUDIT_FORWARD_INTERVAL_MS
  • optional FX_EVAL_RUN_INTERVAL_MS
  • optional FX_SLACK_WEBHOOK_URL

The single-queue and multi-queue overrides are mutually exclusive and accept only names declared in infra/harness/task-queues.yaml. A queue-specific worker starts only the outbox consumer associated with that queue.

The repository consumes published @fabric-harness/sdk@^4.2.4, @fabric-harness/databricks@^4.4.1, and @fabric-harness/temporal@^2.0.1; no sibling checkout or link: dependency is required. The Databricks package owns Unity Catalog Agent Services reconciliation for customer-operated workers.

docker build -t fabric-experiments-worker:staging -f infra/harness/worker.dockerfile .

On startup, verify the log reports all intended queues and the number of created or existing SRM schedules. Deployment is not healthy until Temporal shows a poller for every selected queue and the process emits [harness] ready.