Seven Myths of Business Experimentation
Strategy + Business, 2020-08-10
Some basic stuff first
Innovation is based on business experimentation.
Arguments against experimentation
Perversity – changes will make the company worse, not better
Futility – changes won’t make a dent as they won’t touch on the deeper structural challenges.
Jeopardy [the most dangerous one] – a proposed action involves unacceptable risks and costs. Costs are not hard to estimate, but not the benefits.
The Myths of Experimentation
1. Experimentation-driven innovation kills intuition and judgement
Intuition, customer insights and qualitative research are important sources for hypothesis generation, not immediate implementation.
Experts are poor at predicting customer behaviour.
2. Online experiments won’t lead to breakthrough changes
Incorrect belief that the large change leads to large impact.
Breakthroughs are cumulative and aren’t a result of a single change.
[MK: although focusing on multiple improvements may in fact lead to reaching a local maximum]
A good culture of A/B/x testing, while not very cheap, can lead to building scenarios, which truly have a potential to change user behaviours.
3. We don’t have enough hypotheses for large-scale experimentation
No need to copy Google / Amazon / Booking.com in trying to replicate the number of experiments. [MK: a button moved 3 pixels left is also an experiment]
The need for experiments must come from within.
4. Not enough transactions for experiments in Brick-and-mortar companies
Can’t choose 50k customers at random, have to be limited by individual stores or geographies.
The larger the effect expected – the smaller the sample has to be, and vice versa (to identify the weak signal in the noise).
Large samples are often clustered or correlated, so the true sample is smaller.
Maybe just the process of experimenting will have enough questions asked to modify direction.
When moving from offline to online [Categories 1, 2] experimentation ability is essential.
5. Experiments had only modest effects on the business performance
Problem: the actual sum of improvements is smaller than the expected one.
That’s because interaction effects are not additive (some cancel each other).
ROI is not predictable, so requires a leap of faith and adjustment of models.
6. Understanding causality is not needed in the age of AI
A cliché: correlation is not causation. Finding similarities doesn’t win the war.
But this should lead to the emergence of hypotheses that can be tested.
Models built on big data may be efficient for predictions based on small data sets.
7. Running experiments on customers without consent is not ethical
Problem: in academia and medicine all experiments on people must be approved by ethics review boards; online this doesn’t happen and is a cause for controversies.
Most companies still run experiments, and many have ethics training [MK: trying to make the resulting process look fair].
LinkedIn: experiments must “not deliver a negative member experience, have a goal of altering member’s mood or emotions, or override existing member’s settings or choices”.
Perhaps sticking to the “not worse-off” scenario is enough.
[MK: some “tests” stating that there are “two rooms left” in a hotel are being reviewed by consumer protection authorities.]