Why you should always test Behavioral Interventions on a small scale first

Behavioral interventions can have a very significant impact on human behavior. And since all human systems are complex, any change can have unintended consequences. For this and the other reasons that follows, it’s critical to include a test on a sample phase within all your behavioral projects. You can see how I use it within my behavioral intervention processes here.

Testing on a small-scale sample can nearly always be done in any contexts, either through digital channels or even physical ones. It allows you to validate several aspects of your intervention that are critical to evaluate before planning a full-scale implementation.

Small-scale testing validates your data collection methodology

You need your behavioral interventions to be effective, and for that you need to be able to measure their efficiency. That’s why when you consider your small-scale test phase, you need to also test for your data collection methods.

Human data collection issues can easily arise when the processes impacted are complex and involving several units or organizations. You may discover flaws in how people operate that were not apparent in the first place but hinders data collection when an actual intervention is launched.

Electronic data collection also needs to be tested and validated.

It can be tempting to leave data validation out of the test phase, but it usually ends in regret.

Take the following case study: the University of Vermont banned the sale of bottled water from its campus in a bid to reduce waste and to drive people to use newly installed water fountains to refill reusable water bottles. They did not run a small-scale test before the total campus ban, nor did they put in place the means to evaluate if their ban was successful. When they needed to evaluate the effects of the ban, they were reduced to studying the shipments of beverage for sale on the campus to try to get some answers, and what they saw was not what they expected.

Small-scale testing validates if the intended effects materialize

If you’re familiar with A/B testing in the digital industry, there is a common saying that says “Test everything!”. Its key message is that you shouldn’t assume beforehand that you know how users will react to a change before you test it. You can have excellent intuitions and a ton of experience, but you won’t be right all the time and maybe not the majority of the time. What matters is that you recognize you are fallible and testing is easy to do to validate your thinking.

When designing a behavioral intervention, you should be as thorough as possible but the only real test is the small-scale test phase. A good example here is when the City of Austin banned single-use plastic bags in supermarkets. They allowed supermarkets to only offer sturdier reusable plastic bags. They did not test the ban on a couple of supermarkets first or on a neighbourhood but went straight with the total city ban. When they ran an assessment two years later, they found that the total plastic waste in the city recycling centers had increased: clients were using as many plastic bags as before, they just treated the sturdier reusable bags as single-use, thus increasing the volume of waste from those bigger, thicker bags.

Running a small test phase in a single supermarket and directly counting the number of reusable bags effectively used, they would have detected this phenomenon before wide-scale implementation and tried to counteract its effect. The end result of not testing on a small-scale is that the ban directly increased plastic waste, the complete opposite of its goal.

Small-scale testing lets you identify any unintended consequences

As important as validating intended effects is to identify any possible negative unintended effects you are having on people behavior. This is trickier in practice though, since you may not know where unintended consequences will materialize and that makes it difficult to look for them.

A best practice for any behavioral intervention is to have a set of “control metrics”: these are metrics that identify behavior that could change—for the worse—with your change. Brainstorm what these could be and monitor them on a small-scale test. If we go back to the ban on the sale of bottled water quoted above, after assessing the data they could find after 18 months, they saw that after the ban went in place, sales of the remaining—less healthy—bottled beverages increased strongly. People were looking to buy a bottle of beverage, and if no water was available, they didn’t use the fountains to refill reusable water bottles, they just bought what was on offer: soft drinks and juices. Talk about negative unintended effects…

Testing is an essential step in many, many industrial processes for a reason: you discover things, and sometimes critical issues. Behavioral interventions have at least as much impact as industrial processes when they go wrong and testing should not be viewed as optional.