Take one thing off the first screen.
More players walk in.

The Honest Caveat & The Lesson

Only the activation read at the top had the volume to trust. Everything downstream — Floor 2, finishing the run, coming back the next day — is drawn from ~130 players per arm: far too few to decide anything. We show them for honesty, not as proof. And the real lesson of this log isn't “removing the seal won.” It's this: at our traffic, most experiments can never gather enough data to be reliable — and dressing up a thin, noisy number as a finding is how you fool yourself. When the data can't decide, decide with an informed gut, and make a bet.

Before running any A/B test, put it through these rules (now baked into our internal game-council). If a test fails them, don't run it — trust your gut and ship a bet instead: your best judgment, live, with a guardrail to watch.

1. 1
   Run the power check first — the “can this even be measured?” math
   Work out up front how many players per arm you'd need to actually see the effect. If our funnel can't deliver that within a few weeks, the test is a mirage — you'll collect noise and read tea leaves. Know the answer before you start, not after.
2. 2
   Don't test retention — it sits at the far end of the funnel
   Next-day return is ~4%: a rare event that needs thousands per arm — months of full traffic — to move a single point. Watch it as a health trend; never A/B it. Test something early and common instead.
3. 3
   Test the highest-rate proxy — the earliest, most common step
   Statistical power comes from a high base rate. Read the change on Floor-2 reach (~60%) or the very first tap — not the low, downstream outcome. A common step needs a fraction of the sample a rare one does.
4. 4
   Fewer arms — control plus one bold change
   Every extra arm splits the traffic thinner and demands a bigger sample, and the “winner” of four noisy arms is usually just the luckiest. Two arms halves the wait and the self-deception.
5. 5
   Only swing big — a bold bet, not fine-tuning
   A ~+10pp swing on a common proxy is testable in days; a few-point tweak never is at our size. If the honest expected effect is small, don't test it — make it a bet: ship your best call and watch a guardrail. Big bats, or gut and go.

The through-line: low throughput means you cannot buy certainty with a test. Spend your rare tests only on big, early, two-armed, powered questions — and decide everything else on gut feeling, shipped as a bet.