Seven Myths of Business Experimentation

Strategy + Business, 2020-08-10

Some basic stuff first

  • Innovation is based on business experimentation.

Arguments against experimentation

  • Perversity – changes will make the company worse, not better

  • Futility – changes won’t make a dent as they won’t touch on the deeper structural challenges.

  • Jeopardy [the most dangerous one] – a proposed action involves unacceptable risks and costs. Costs are not hard to estimate, but not the benefits.

The Myths of Experimentation

1. Experimentation-driven innovation kills intuition and judgement

  • Intuition, customer insights and qualitative research are important sources for hypothesis generation, not immediate implementation.

  • Experts are poor at predicting customer behaviour.

2. Online experiments won’t lead to breakthrough changes

  • Incorrect belief that the large change leads to large impact.

  • Breakthroughs are cumulative and aren’t a result of a single change.

  • [MK: although focusing on multiple improvements may in fact lead to reaching a local maximum]

  • A good culture of A/B/x testing, while not very cheap, can lead to building scenarios, which truly have a potential to change user behaviours.

3. We don’t have enough hypotheses for large-scale experimentation

  • No need to copy Google / Amazon / in trying to replicate the number of experiments. [MK: a button moved 3 pixels left is also an experiment]

  • The need for experiments must come from within.

4. Not enough transactions for experiments in Brick-and-mortar companies

  • Can’t choose 50k customers at random, have to be limited by individual stores or geographies.

  • The larger the effect expected – the smaller the sample has to be, and vice versa (to identify the weak signal in the noise).

  • Large samples are often clustered or correlated, so the true sample is smaller.

  • Maybe just the process of experimenting will have enough questions asked to modify direction.

  • When moving from offline to online [Categories 1, 2] experimentation ability is essential.

5. Experiments had only modest effects on the business performance

  • Problem: the actual sum of improvements is smaller than the expected one.

  • That’s because interaction effects are not additive (some cancel each other).

  • ROI is not predictable, so requires a leap of faith and adjustment of models.

6. Understanding causality is not needed in the age of AI

  • A cliché: correlation is not causation. Finding similarities doesn’t win the war.

  • But this should lead to the emergence of hypotheses that can be tested.

  • Models built on big data may be efficient for predictions based on small data sets.

7. Running experiments on customers without consent is not ethical

  • Problem: in academia and medicine all experiments on people must be approved by ethics review boards; online this doesn’t happen and is a cause for controversies.

  • Most companies still run experiments, and many have ethics training [MK: trying to make the resulting process look fair].

  • LinkedIn: experiments must “not deliver a negative member experience, have a goal of altering member’s mood or emotions, or override existing member’s settings or choices”.

  • Perhaps sticking to the “not worse-off” scenario is enough.

  • [MK: some “tests” stating that there are “two rooms left” in a hotel are being reviewed by consumer protection authorities.]