Can your traffic support a useful test?

For one fixed-horizon comparison of two independent groups, with a yes/no conversion outcome and a 50/50 random split. Count each eligible person once. Use a stable baseline from comparable traffic.

Agree the primary metric, allocation, exclusions and decision rule before launch. Wait for the planned sample and mature outcomes; repeatedly checking significance can increase false positives.

Geo tests, repeated exposures, revenue-per-user metrics and multiple variants need a different design. This estimate uses a normal approximation without a continuity correction; rare outcomes may need specialist sizing.