Why I Run Experiments

On trading opinions for evidence.

Most decisions I've watched go wrong didn't fail for lack of intelligence. They failed because a confident guess was allowed to stand in for a fact. Someone had a strong opinion, the opinion sounded reasonable, and nobody paid the small cost of checking whether it was true. Experiments are how I pay that cost on purpose.

An experiment is just a question asked honestly enough that reality can answer it. You state what you believe, you decide in advance what would change your mind, and then you go and look. That last part — deciding in advance — is the whole game. It's what separates an experiment from a demo designed to confirm what you already wanted to do.

Opinions are cheap; evidence compounds

I run experiments because opinions don't accumulate and evidence does. Ten people with ten strong opinions leave you exactly where you started: arguing. Ten small experiments leave you with ten things you now actually know, and those facts stack on top of each other. Over months, the difference between a team that guesses and a team that tests isn't a little bit of accuracy. It's the difference between learning and merely having meetings.

Being wrong quickly is the point

There's a quiet fear underneath a lot of decision-making: that being wrong is embarrassing. Experiments reframe that. If the goal is to find the truth, then a result that kills your favourite idea isn't a failure — it's the experiment doing its job, cheaply, before the idea cost you a quarter. I'd rather be wrong on a Tuesday afternoon with a small test than be wrong in production with everyone watching.

The habit matters more than any single result

No single experiment is going to be profound. The value is in the habit: reaching for evidence by reflex, writing down what you expect before you look, and treating "I don't know yet, let's find out" as a perfectly respectable thing to say out loud. That habit is what I'm trying to build, in myself and in the way I work.

That's what Experiment Forge is about. More soon.