What Makes a Good Experiment

A few things I try to get right before I start.

Not every test is worth running, and not every test that gets run actually teaches you anything. Over time I've collected a short list of things that separate an experiment that moves you forward from one that just burns a week and confirms your bias. None of it is complicated. All of it is easy to skip.

Start with a question you can be wrong about

If there's no result that would change your mind, you don't have an experiment — you have a plan you're looking to justify. The first thing I write down is the belief I'm testing, stated plainly enough that reality could contradict it. If I can't imagine the outcome that would prove me wrong, I'm not ready to start.

Decide what "success" means before you look

The most common way experiments go sideways is moving the goalposts after the results are in. So I pick the metric and the threshold up front: what am I measuring, and what number counts as a yes? Writing it down beforehand takes five minutes and saves you from the very human urge to reinterpret a disappointing result into a win.

Change one thing at a time

If you change five things and the number moves, you've learned that something worked, which is almost useless — you can't repeat it or build on it. The tighter the change, the cleaner the lesson. Sometimes that means a smaller, less exciting experiment. That's usually the right trade.

Make it cheap enough to actually run

An experiment you keep postponing because it's too big teaches you nothing. I'd rather run the scrappy version this week than the perfect version never. The goal isn't a beautiful study; it's a fast, honest answer that lets you decide what to do next.

Write down what you learned — especially when it's boring

The result is only half the value. The other half is the record: what you expected, what happened, and what you'd do differently. A "no" that's written down is a fact your future self and your team get to keep. A "no" that's forgotten just gets rediscovered the expensive way, later.

Good experiments aren't clever. They're honest, small, and written down. More soon.