Databricks Harness worker

Fabric Experiments can run its durable Fabric Harness agents as a separate Databricks App in the customer's workspace. The App is the execution plane; the Experiments App remains the control plane. Keeping them separate gives the worker its own service principal, secret grants, SQL warehouse grant, model endpoint, logs, and release lifecycle.

The worker currently runs three certified agents:

• SRM watcher — scans running experiments on a five-minute Temporal schedule;
• Experiment analyst — produces evidence-bound readouts from terminal experiment events; and
• Evaluation runner — executes bounded generation and judge work and writes Quality evidence.

Set FX_EVAL_DISPATCH=worker on the Experiments control App only after the worker is deployed and ready. In this mode, creating an evaluation commits the run and outbox row atomically; the worker starts the Temporal workflow and acknowledges the row only after Temporal accepts it. If the worker is absent, new evaluations remain safely pending rather than falling back to inline execution.

These are immutable definitions compiled into the official worker. Experiments stores their version, implementation digest, runtime version, model policy, capability grant, status, and run attestations. The worker must receive an admission response before an agent body executes. A disabled registration or digest/runtime mismatch fails closed. Only SHA-256 input/output hashes cross the run-control boundary; prompts, examples, and readouts are not copied into the control-plane run record.

The current built-in definitions are version 3 and advertise the exact runtime @fabric-harness/sdk@4.2.4. The standalone build compiles this shared admission contract before bundling the worker and rejects an artifact containing legacy Harness 4.1 runtime metadata.

Startup installs missing certified registrations and advances existing registrations to the worker's newer immutable built-in version while preserving an administrator's disabled status. A changed definition that does not increment its numeric version is rejected as a release error.

Prerequisites

• a deployed Fabric Experiments App and organization ID;
• a Databricks SQL warehouse, analytics schema, and a separate Unity Catalog telemetry schema backed by a custom managed storage root;
• a Temporal Cloud namespace reachable from Databricks Apps over TLS;
• an expiring, write-scoped Experiments API key for exactly one organization;
• the same worker-api-token configured on the Experiments App; and
• a Databricks Model Serving endpoint for evaluation generation and judges.

For customer-agent Unity Catalog enrollment, also enable the Agent Services Beta in the workspace, create a UC HTTP connection for the agent's external Responses endpoint, and grant the worker service principal permission to create Agent Services and manage their grants in the selected schema.

Create these keys in the configured Databricks secret scope:

• harness-fx-api-key
• worker-api-token
• temporal-address
• temporal-namespace
• temporal-api-key

The worker App receives CAN_USE on the selected SQL warehouse and READ on only those secrets. It does not receive the Experiments database credential.

Validate and deploy

The worker is a standalone Asset Bundle because Databricks permits only one App to own a given uploaded source path inside a bundle. Build the minimal App artifact first. It contains the compiled worker, supervisor, task-queue declaration, Databricks app.yaml, and a standalone package manifest with published Harness, Temporal, and OpenTelemetry versions; the bundle does not upload the monorepo or any workspace:/link: dependency. From the repository:

pnpm install --frozen-lockfile
pnpm build:databricks:worker

cd deploy/databricks/harness-worker

databricks bundle validate -t prod \
  -p fabric-harness \
  --var='control_app_url=https://<experiments-app>.databricksapps.com,warehouse_id=<warehouse-id>,analytics_catalog=<catalog>,analytics_schema=<schema>,telemetry_schema=<custom-storage-schema>,org_ids=<org-id>,model_endpoint=<serving-endpoint-name>,worker_build_id=<git-sha>'

databricks bundle deploy -t prod \
  -p fabric-harness \
  --var='control_app_url=https://<experiments-app>.databricksapps.com,warehouse_id=<warehouse-id>,analytics_catalog=<catalog>,analytics_schema=<schema>,telemetry_schema=<custom-storage-schema>,org_ids=<org-id>,model_endpoint=<serving-endpoint-name>,worker_build_id=<git-sha>'

databricks bundle run harness_worker \
  -t prod \
  -p fabric-harness \
  --var='control_app_url=https://<experiments-app>.databricksapps.com,warehouse_id=<warehouse-id>,analytics_catalog=<catalog>,analytics_schema=<schema>,telemetry_schema=<custom-storage-schema>,org_ids=<org-id>,model_endpoint=<serving-endpoint-name>,worker_build_id=<git-sha>'

Use bundle run, not a direct databricks apps deploy, so Databricks applies the deployment-specific App config from the bundle. The generated build/app.yaml supplies the command and resource-backed defaults; the bundle adds the control-plane URL, immutable tenant binding, analytical destination, and build identity. A direct deploy intentionally lacks those tenant-specific values and therefore fails closed.

Fabric release certification can build an unpublished exact Harness candidate without publishing it first:

FX_HARNESS_CANDIDATE_ROOT=../fabric-harness pnpm build:databricks:worker

That mode vendors the SDK, Databricks integration, and Temporal tarballs and records the Harness commit in the generated private package. Do not distribute that candidate artifact to customers. Omit the variable for customer builds; the generated manifest then pins the corresponding public registry versions.

model_endpoint is the serving endpoint name accepted by /api/2.0/serving-endpoints/{name}/invocations (for example, databricks-llama-4-maverick), not a Unity Catalog model identifier such as system.ai.gpt-oss-20b. The bundle declares that endpoint as an App resource and grants the worker service principal only CAN_QUERY.

Databricks Apps telemetry does not accept tables in workspace default storage. Create telemetry_schema with a customer-owned MANAGED LOCATION contained by a Unity Catalog external location before deploying the worker. The App runtime captures stdout and stderr into harness_worker_otel_logs. The supervisor also uses the Databricks-provided OTLP endpoint to emit fabric.harness.worker.alive and fabric.harness.worker.ready gauges plus health and lifecycle spans into harness_worker_otel_metrics and harness_worker_otel_spans. Local runs remain uninstrumented unless OTEL_EXPORTER_OTLP_ENDPOINT is configured.

Set worker_build_id to a release or commit digest so each run attestation identifies the executable build.

Grant the worker App's service principal CAN_USE on the Experiments control App. This Databricks App-to-App grant is separate from the Fabric API key and worker token. Requests carry Databricks OAuth at the gateway, then both the internal worker token and API key at the application boundary. The worker token authenticates the process class but never selects a tenant. The independently revocable FX_API_KEY binds every bootstrap, run-admission, completion, and Agent Services reconciliation operation to its one organization; a body that names any other organization is rejected.

control_app_url is the App origin (for example, https://fabric-experiments-prod-<workspace-id>.<region>.databricksapps.com), without an /api suffix. Internal worker routes live under /internal; public tenant routes live under /api/v1.

Verify

The App supervisor exposes:

• /healthz when the supervisor process is alive; and
• /readyz only after the Harness worker connects, starts every selected Temporal poller and consumer, and registers the SRM schedules.

Databricks Apps OAuth may protect both endpoints before the request reaches the supervisor. Verify the App log contains [harness] ready, then confirm the Harness agents Studio page shows all three registrations enabled and new runs as Temporal executes them. Because Databricks applies telemetry settings to the running App snapshot, redeploy after changing any telemetry destination. Then query all three prefixed tables and require recent rows for the worker App service name before treating observability as certified.

During a redeploy, the App supervisor and Temporal SDK can observe the same SIGTERM. The worker treats Temporal's existing DRAINING or STOPPED state as an idempotent close, waits for the run loop to settle, and still surfaces every other shutdown error. A clean replacement log must show the drain transitions without an IllegalStateError before the new [harness] ready marker.

Run one worker instance per complete queue set initially. If you split queues, use FX_HARNESS_TASK_QUEUE or FX_HARNESS_TASK_QUEUES and ensure exactly the intended processes run the schedule registrar and outbox consumers.

Customer-authored agents

The certified worker does not download or evaluate customer code. Build the agent with Harness defineAgent() and deploy its executable under a separate customer-owned identity. Experiments enrolls the executable by immutable digest; it never treats enrollment as certification of arbitrary source code.

Enrollment is a two-step admin operation:

1. register the tenant's raw 32-byte Ed25519 public key; and
2. submit a signed manifest whose tenantId matches the active organization.

fx harness register-key --input '{
  "keyId":"customer-prod-2026-07",
  "publicKeyHex":"<64-lowercase-hex-characters>"
}'

fx harness enroll --input "$(cat signed-agent-envelope.json)"

# Prevent this key from authorizing any later enrollment.
fx harness revoke-key customer-prod-2026-07

Use canonicalCustomerHarnessAgentManifest() from @fabricorg/experiments-api-protocol as the bytes signed with Ed25519. The envelope contains manifest and signature; the signature contains keyId, alg: "ed25519", the 128-character hexadecimal signature, and the 64-character SHA-256 signedHash. Agent IDs must begin with customer:.

When databricksAgentService is present, the manifest also supplies a three-part fullName, the existing UC HTTP connection name (not a REST resource path), an absolute basePath such as /responses, and one or more executePrincipals. The worker atomically claims the pending version, uses the published @fabric-harness/databricks Agent Services client to create or verify it, applies EXECUTE grants, and reports ready for that exact version. It fails closed rather than taking over an existing service with a different connection or base path.

Set FX_AGENT_SERVICES_RECONCILE=1 only after acknowledging the Databricks Beta. Run a separate worker deployment and API key for each Fabric organization; multi-tenant reconciliation requests are rejected. A failed reconciliation is visible in the Harness control plane and does not silently become ready. A newer manifest must increment version; reusing a version with a different digest or configuration is rejected.

For a transient Databricks failure, an administrator can set the registration to enabled again through the registrations API, or disable and re-enable it in Studio. That queues the same signed version for reconciliation without changing its manifest, digest, or key material.

Agent Services currently provides registration, discovery, metadata, and permissions for external agents. Databricks does not expose runtime invocation through the Beta, so Experiments tests execution separately through worker run admission and attestations.

Start with Fabric Harness's Your First Agent walkthrough, then use Agent anatomy for policies, tools, skills, schemas, and strict-mode definitions. For a customer-owned Databricks runtime, continue with the Fabric Harness Databricks guide.