Guardrails
SRM, metric decline, and error-rate checks protect experiments while they run.
Guardrails evaluate whether an experiment should continue running.
The SRM watcher uses the same guardrail evaluation code as the API. It compares
observed exposure counts with the experiment's configured variant weights using
Pearson's chi-squared test. Evaluation begins at 100 total exposures. The
default breach threshold is p < 0.001; 0.001 <= p < 0.1 is reported as
near-threshold and does not change experiment state.
The production Harness workflow evaluates running experiments every 5 minutes.
On a breach it records a typed guardrail.breached audit event and invokes
experiment.guardrail_breach with action pause. A running experiment moves
to paused; duplicate or late signals cannot move an experiment from an
incompatible state. The operator job is dry-run by default and mutates state
only when apply: true is explicitly supplied.
Common guardrails:
- SRM — sample ratio mismatch; detects assignment or delivery imbalance.
- Metric decline — detects harmful movement in a key metric.
- Error rate — detects variant execution or client-side failure spikes.
For metric_decline, the configured threshold is the relative drop from the
control and a breach also requires a two-proportion p-value below 0.05. For
error_rate, the threshold is the fraction of exposures carrying a failure.
These guardrails use the same pure evaluator, but only SRM is scheduled by the
built-in watcher today.
Set FX_SRM_THRESHOLD to override the SRM p-value threshold and
FX_SRM_METRIC to select the aggregate metric read during the scan. Keep the
default unless your experiment review policy records and approves a different
false-positive tolerance.