Single-Variable Testing

Change one variable per arm so the winner is attributable. Otherwise it's a guess.

How to apply it

If you must test combinations, use a multivariate (MVT) design with proper sample size.

Single-Variable Testing in practice: a concrete example

The test that can't tell you what won

You change the button colour to green and rewrite its label in the same variant, and conversion rises 9%. Which change did it? You can't say, because you moved two things at once, so the “winner” teaches you nothing you can reuse. Change one variable per arm and the result is attributable: the lift belongs to the thing you changed. If you genuinely need to test combinations, that is a multivariate test with a proper design and sample size, not a two-change guess.

Common Single-Variable Testing mistakes

Bundling several changes into one variant, so a win or loss can't be traced to any single cause.
Calling a confounded test a success and rolling the whole bundle out, carrying along changes that may have hurt.
Reaching for multivariate testing without the traffic to power all the combinations, which just spreads the sample too thin.

When Single-Variable Testing doesn't apply

Strict one-variable isolation is for learning why something worked. When you simply need the best of a few fully-formed designs and don't need to attribute the cause, a straight A/B between complete variants is fine. And a coherent redesign is often best shipped and measured as a whole, since splitting it into isolated variables would test a product nobody would actually build.

Practise it on a real challenge

Hero Test — three things changed at once

A team ships 3 variants of a hero with different copy, color, and image.

Single-Variable Testing

How to apply it