exd telemetry simulate

Re-evaluate historical evaluation records against a candidate manifest. "If I push this, how would last week's traffic have looked?"

Synopsis

exd telemetry simulate --manifest <path>
    [--flag <key>]...
    [--namespace <slug>] [--environment <env>]
    [--since <duration> | --from <ts> --to <ts>]
    [--source <uri>]...
    [--format human|json]
    [--engine <name>]
    [--fail-on-error]

See common flags for shared options.

Description

For each historical evaluation in the source, re-evaluates the same context against the candidate manifest and reports the new variant distribution. This is the unique exd capability enabled by manifest-version provenance — a candidate manifest can be tested against real production traffic before it is pushed.

The candidate MUST pass exd lint. The historical source MUST contain context_attributes (records produced under namespaces with all attributes private will lack the data needed for re-evaluation).

Use cases

• Pre-push impact analysis. Before pushing v14 to production, simulate it against the last 24 h of real evaluations:

  exd telemetry simulate --manifest ./namespaces/marketing --flag onboarding-banner --since 24h

  Output shows the candidate's variant split vs. the observed split — the delta is the change real users will experience after push.

• Validate a complex rule rewrite. Restructure a rule chain; simulate; confirm no unintended audience flip:

  exd telemetry simulate --manifest ./candidate --since 7d

• A/B-test the test plan. If your rollout would have created a 50/50 split last week, but simulate shows 70/30, your rule audience overlaps with another flag's segment in a way you didn't expect.

• Approve a rollout by data. A blocker for shipping ambitious changes is reviewer uncertainty. simulate produces a concrete number: "v14 changes 0.2% of evaluations." Easier to ship.

Subcommand-specific flags

Flag	Required	Notes
--manifest <path>	yes	Local path to the candidate manifest directory. Must pass exd lint.

Result fields (--format json)

Field	Type	Notes
observed.variants	map	Observed variant counts in the historical window.
simulated.variants	map	Variant counts under the candidate manifest.
delta	map	Per-variant simulated - observed.
changed_evaluations	integer	Count of evaluations whose variant changed.
changed_pct	float	changed_evaluations / total.
flag_breakdown	array	Per-flag rows of the same shape.

Diagnostics

• T011 private-attribute leakage (if the historical records leaked, the simulate path inherits)
• Any diagnostic the candidate's lint pass would emit (as a load failure → exit 1)

Example

$ exd telemetry simulate --manifest ./candidate --flag onboarding-banner --since 24h

Simulating ./candidate against historical traffic for 'onboarding-banner' (last 24h, n=198,432)

variant       observed     simulated      delta
treat_a        91,008        98,118     +7,110
treat_b        90,521        90,498        -23
control        16,903         9,816     -7,087

changed_evaluations: 14,197 (7.2%)

The candidate would move ~7,100 evaluations from 'control' to 'treat_a' — primarily users
where the new rule[1] would now match (predicate widened from US → US,CA).

Exit codes

See telemetry exit codes. Candidate-manifest lint errors → exit 1.

See also

• exd lint — runs the same pipeline; simulate refuses to run with errors.
• summary — the observed half of simulate's output, in isolation.
• exd manifest push — push the candidate once you're happy with the simulation.