Skip to main content

5. Roll out with the testing attribute

← Previous: Fixtures and automated tests · Index · Next: Troubleshoot with exd eval

By the end of Chapter 4 you had a flag whose behavior is lint-clean, schema-aligned with the application code, and pinned by a fixture-driven test. Time to ship it — but not all at once. The bucket math is mechanical, the predicate is one eq, the test suite passes — and yet running A/B/C in production for the first time always has a non-zero chance of biting. You want to run the new rules end-to-end in production with a small audience before flipping them on for everyone.

That's what the testing = true attribute does. It lets a flag's per-env rules execute only for callers that explicitly opt in — typically a dev running an admin UI, or an automated end-to-end test driven by a CI bot with the opt-in baked in. Regular traffic continues to see the env's variant (the off-state) for as long as the gate is on.

This chapter walks through the canonical three-state rollout: Disabled → Testing → Enabled. Each state is a specific env-block shape — there's no separate state field, the shape is the state. The spec reference is docs/reference/manifest/09-resolution.md § The rollout workflow.

Why a new [flag.environments.production] block?​

In Chapter 1 you put the bucket rules under [flag.environments._] — the catch-all. Every env that doesn't declare its own block walks those rules. For the rollout we add a production-specific block on top. Production walks the new block; staging, development, and any other env keep falling through to _.

The reason: testing = true is only legal on named env blocks. The catch-all _ cannot be testing-gated (lint emits E039 if you try). And the whole point of the rollout workflow is to differentiate one env's behavior from the others — production gets the new rules, the rest don't, until you're ready.

Day 0 — Disabled in production​

The first edit adds the production env block in its kill-switch shape: a variant and no rules.

marketing/flags/welcome-banner.toml — append this block:

# Day 0 — disabled in production. Step 2 returns `control` immediately; the
# catch-all _'s rules are never consulted because step 2 short-circuits.
[flag.environments.production]
variant = "control"

Now confirm the new state with exd explain:

$ exd explain welcome-banner --env production --manifest marketing
...
=== Resolution walk
[flag.environments.production].rules                empty
[flag.environments.production].variant = control
[flag.environments._].rules                         (not reached — step 2 returned)
[flag.environments._].variant                       (not reached)

Step 2 short-circuits. Every production caller gets control, regardless of user.id or user.country. Other envs still walk _'s buckets via fallthrough — you can confirm by switching --env:

$ exd explain welcome-banner --env staging --manifest marketing
...
=== Resolution walk
[flag.environments.staging].rules                   SKIPPED — env did not declare its own
[flag.environments.staging].variant                 SKIPPED — env did not declare its own
[flag.environments._].rules
  rule[0] segment `welcome-banner-bucket-control`   variant: control
  rule[1] segment `welcome-banner-bucket-treat-a`   variant: treat_a
  rule[2] segment `welcome-banner-bucket-treat-b`   variant: treat_b
[flag.environments._].variant = control

Staging sees the buckets; production sees control. The flag is now safely cordoned.

Day 1 — Testing in production​

Time to roll out to admins. Two edits to the production block: add testing = true, and copy in the three bucket rules.

# Day 1 — testing in production. Internal admins and CI bots opt in via
# `include_testing = true` on the eval call; everyone else still sees `control`.
# The catch-all _'s rules stay skipped because production now declares its own
# rules (step 3 short-circuit).
[flag.environments.production]
testing = true
variant  = "control"

[[flag.environments.production.rules]]
description = "Bucket 1: control (US only)"
segment     = "welcome-banner-bucket-control"
variant     = "control"

[[flag.environments.production.rules]]
description = "Bucket 2: treatment A (US only)"
segment     = "welcome-banner-bucket-treat-a"
variant     = "treat_a"

[[flag.environments.production.rules]]
description = "Bucket 3: treatment B (US only)"
segment     = "welcome-banner-bucket-treat-b"
variant     = "treat_b"

Run lint and re-explain:

$ exd lint marketing/
namespace 'marketing': 1 flag, 3 segments — OK

$ exd explain welcome-banner --env production --manifest marketing
...
=== Resolution walk
[flag.environments.production].rules
  rule[0] segment `welcome-banner-bucket-control`   variant: control    (testing-gated)
  rule[1] segment `welcome-banner-bucket-treat-a`   variant: treat_a    (testing-gated)
  rule[2] segment `welcome-banner-bucket-treat-b`   variant: treat_b    (testing-gated)
[flag.environments.production].variant = control
[flag.environments._].rules                         SKIPPED — env declared its own rules
[flag.environments._].variant                       (not reached)

=== Pitfalls
  [catch_all_rules_skip] this flag has 3 shared rule(s) that apply to every other env — but they will NOT run in `production`, because `production` has its own 3 rule(s)

The pitfall fires because production declares its own rules — _'s rules are skipped in production. For this rollout that's exactly what you want: production runs only the testing-gated rules; other envs still walk _'s.

Opt in from the SDK​

The opt-in is per eval call, not a token scope or namespace setting. The same admin user evaluating the flag without the opt-in still sees control. This keeps the opt-in scoped to the explicit UI surface (typically an internal admin tool) and prevents accidental leakage through unrelated code paths.

Rust SDK — use the _with_options variant and pass EvalOptions::new().with_include_testing(true):

use exd_client::{EvalContext, EvalOptions, Namespace};

let ns = Namespace::from_dir("marketing")?;
let admin_ctx = EvalContext::builder()
    .str("user.id",      "u-admin-42")
    .str("user.country", "US")
    .build();

// Admin tool: opted-in. Walks the testing-gated rules → returns treat_a/b/control
// per bucket.
let canary = ns.eval_string_with_options(
    "welcome-banner",
    "production",
    &admin_ctx,
    &EvalOptions::new().with_include_testing(true),
)?;

// Regular handler: same context, no opt-in. Step 1 skipped → step 2 returns "control".
let normal = ns.eval_string("welcome-banner", "production", &admin_ctx)?;

TypeScript SDK — pass { includeTesting: true } on the eval call:

// Admin tool path.
const canary = client.evalString(
  "welcome-banner",
  "production",
  { "user.id": "u-admin-42", "user.country": "US" },
  "",
  { includeTesting: true },
);

// Regular path: no opt-in → sees control.
const normal = client.evalString(
  "welcome-banner",
  "production",
  { "user.id": "u-admin-42", "user.country": "US" },
  "",
);

ExdRemote and server POST /evaluate — the same includeTesting field on the request body / call options.

Probe with exd eval​

Before you build the admin UI, sanity-check both paths from the shell. exd eval doesn't currently accept --include-testing as a flag — but you can simulate the opted-in walk with --trace, which reports which step fired:

$ exd eval welcome-banner --env production --manifest marketing \
    --ctx user.id=u-1 --ctx user.country=US --trace
[flag.environments.production].rules                SKIPPED — caller did not opt into testing
[flag.environments.production].variant = control    MATCHES        → control
[flag.environments._].rules                         SKIPPED — production declared its own rules
[flag.environments._].variant                       (not reached)

outcome: control
  via:   env `production` variant (testing rules were skipped — caller did not pass include_testing)

That confirms the off-state for regular traffic. To exercise the gated path, run the admin-side application code (or the equivalent SDK test) — the typed-eval ..._with_options variants are the only path that carries the opt-in.

End-to-end testing in production​

This is the point of the chapter. While testing = true is on:

  • Admins running the internal UI with include_testing = true see real variants and can verify the banner copy renders, the analytics pipeline records the variant exposure, the downstream service routes the user correctly, etc. They're real production users hitting real production code, just with a private feature toggle.
  • Regular users continue to see control because their eval calls don't pass the opt-in. Whatever bug might be lurking in the new buckets can only surface for the opted-in audience.
  • CI bots with an include_testing = true integration test run against production (or a production-shaped staging env) and prove the new rules behave the same end-to-end as they did against the local fixture suite from Chapter 4.

Stay in this state until you're satisfied. Days, weeks — there's no spec-enforced ceiling.

Day 7 — Enabled in production​

Once admins and CI report happy, flip the gate off. Only one edit: drop testing = true. Same rules, same variant, same segments.

# Day 7 — enabled. Now every production caller walks the rules.
[flag.environments.production]
variant = "control"

[[flag.environments.production.rules]]
description = "Bucket 1: control (US only)"
segment     = "welcome-banner-bucket-control"
variant     = "control"

[[flag.environments.production.rules]]
description = "Bucket 2: treatment A (US only)"
segment     = "welcome-banner-bucket-treat-a"
variant     = "treat_a"

[[flag.environments.production.rules]]
description = "Bucket 3: treatment B (US only)"
segment     = "welcome-banner-bucket-treat-b"
variant     = "treat_b"

Re-explain:

$ exd explain welcome-banner --env production --manifest marketing
...
=== Resolution walk
[flag.environments.production].rules
  rule[0] segment `welcome-banner-bucket-control`   variant: control
  rule[1] segment `welcome-banner-bucket-treat-a`   variant: treat_a
  rule[2] segment `welcome-banner-bucket-treat-b`   variant: treat_b
[flag.environments.production].variant = control

The (testing-gated) annotation is gone. Every production caller now walks the bucket rules.

Optional cleanup: collapse the duplicate​

At this point, the production env block and the _ catch-all carry identical rules. Two ways to clean up:

  1. Leave it. No correctness issue; the production block fires first, and the duplicate exists. Reviewers see exactly which rules apply where.
  2. Delete the production block. Production falls through to _ like every other env. The flag returns to the shape from Chapter 1. Smaller diff, no duplication.

Pick option 2 when every env eventually runs the same rules (the welcome-banner case). Pick option 1 when production-specific tweaks are likely to land later — keep the block as a place for them to go.

Lint guardrails worth knowing about​

Two diagnostics gate the rollout workflow:

  • E039 — testing = true requires rules. The gate has to have something to gate. testing = true paired with an empty rules array is almost always an editing mistake (you removed the rules but forgot to drop the gate, or you added the gate before you authored the rules). Lint blocks the commit.
  • E039 on _ — the catch-all cannot be testing-gated. Audiences come from non-opted-in callers in other envs, so a testing-gated _ would have no audience.

Both fire from exd lint directly — no separate command, no extra setup. Chapter 2's CI gate already covers them.

Next​

Chapter 6 — Troubleshoot with exd eval. The flag is now live; the last chapter is what to do when a real user calls support to say "the banner looks wrong."