How to test one change at a time

Everyone is told to experiment. Almost nobody does it in a way that produces knowledge, because they change several things simultaneously and then interpret whatever happens.

A test that teaches you something has one variable, a prediction made in advance, and enough repetitions to mean anything.

Change one thing

This is the entire method and it is the part people skip. If the opening, the length and the subject all changed, the result cannot tell you which mattered.

Hold everything else as steady as you can. It feels slow. It is the only version that compounds.

Predict first

Write down what you expect before you post. "I think leading with the result will hold more people past five seconds."

Predicting in advance prevents the main failure of informal testing, which is explaining whatever happened after the fact. If you did not say it beforehand, you will find a reason afterwards regardless of the outcome.

Pick the measure that matches the change

A change to the opening should be judged on early retention, not total views. A change to the ending should be judged on completion or rewatches. A change to the caption should be judged on where the traffic came from.

Judging every change by view count guarantees noise, because view count moves for a dozen unrelated reasons.

Repeat it

One video each way is not a test. Variation between individual videos is larger than most of the effects you are trying to detect.

Three or four of each is a reasonable minimum, and even then treat a small difference as inconclusive. Large, obvious differences are the only ones worth acting on from small samples.

Test things that transfer

Some results apply only to one video. Others apply to everything you make.

Structural questions — does showing the result first work for me, does my audience prefer shorter, do questions mid-video get more replies — produce knowledge you will use for years. Testing a specific joke does not.

Keep a record

Write down what you changed, what you predicted, and what happened. A short note per test.

Without it you will re-run the same experiments every few months and never accumulate anything. The record is what turns a year of posting into a year of learning — and it is five minutes a week.