Our contrary tendency to see experiments—but not untested innovations foisted on us by powerful people—as involving risk, uncertainty, and power asymmetries […]. — source
They summarized their view in a NYT op-ed and Michelle Meyer expands on them both in a forthcoming article (SSRN link) and the following blog entry. I really recommend all 3 of them for more thoughtful info at various levels of details.
In layman’s terms, people tend to be outraged when randomized testing is conducted on uninformed customers (so necessarily without their consent), say between 2 versions of a client dashboard in the case of online banking, but they don’t see any problem if their bank is changing their current dashboard to one of these 2 new versions without doing any testing.
By running an A/B test between the 2 new versions1 on, say, 10% of its clients, the bank could rigorously test if one of them leads to better customer satisfaction than the other. But that requires running a test on uninformed clients who have not given the bank consent to be used as subjects, since informing them of the experiment will bias their views.
If the bank wants to avoid testing, then it will choose 1 of the 2 versions, without doing an A/B test (but surely by doing user research with customer panels, etc.). Since customer panels generate less reliable insights than a proper A/B testing program, the bank could end up implementing the version that leads to lower customer satisfaction.
Skirting the test protects the bank from potential customer backlash but leads to average lower satisfaction for its clients.
So this tendency by customers to negatively view experimentation can prevent entities (public or private) to to conduct randomized testing that would result in higher satisfaction for these same customers.
This A/B illusion is a very real mechanism which will subside at least until consumers understand A/B testing and RCT better, and it has important consequences too.
A/B illusion or A/A illusion?
The one thing I am not fond of about the concept is its name: it seems that illusion applies to the process of running A/B tests, which is of course not the view of the authors.
Naming it the A/A illusion would I think illustrate the concept more clearly. It shows that not testing a new version against the current one but shipping it directly to all customers is akin to running an A/A test. It better frames the view that 1 version is also an experiment, and that people are deluding themselves if they think they are better off avoiding all A/B testing.
Related: behavioral design disclosure statements
I have written about the benefits of disclosing the tests and nudges used by companies and governmental agencies through behavioral design disclosure statements here. This can be considered by companies wary of generating bad PR.
- I’m simplifying of course, since you would want to keep the current one as control, not test entirely new versions, but evolve by repeated testing, etc… ↩