Читать книгу Engaged - Amy Bucher - Страница 34

Real-world comparisons use an existing comparison group instead of a randomized comparison of people using a product to people using something else. Unlike an RCT, you don’t control what programs, if any, the comparison group uses. You do try to establish similarities between the comparison and test groups; for example, you might compare people using your new walking program to people who have a Fitbit. Often, a real-world comparison is much more feasible for companies to do than an RCT. They provide strong evidence for a product as long as the research team takes steps to rule out alternative explanations for results.

Here’s an example from my own work. The members of a large health plan had access to digital health-coaching programs to target a handful of chronic conditions like high blood pressure and diabetes. Some of the members used the programs; others did not. The health plan wanted to figure out if the programs made any difference. So they were able to identify a couple thousand program users in their database and separate them out. Then they identified an equal number of nonusers in their database who were pretty similar to the users at the outset; a similar mix of men and women, average age, and level of disease symptoms. Finally, they compared the data for the two groups at different time points to see if their outcomes progressed the same way or if one group looked different from the other. They found that people who used the programs ended up having less severe disease symptoms (and fewer healthcare expenses) in the years after the baseline data was collected. Because the comparison group was selected to be as similar as possible to the user group, the health plan had confidence that using the programs made a difference in people’s outcomes.

TIP USE A THIRD PARTY

In the example of the health plan study, my company made use of third-party data supplied by our client: their claims database. Third parties can be an excellent source of outcomes data, depending on what your product does. If you don’t have access to data through a client or partner, many companies sell access to third-party databases. This data can be especially useful for filling in some of the gaps about people’s behavior: Pharmacy data can show who is filling prescriptions, consumer data can show who is buying supplies, and financial data can show who is saving money. Just be careful to honor your users’ trust in using third-party data; it can be easy to cross the line from inquisitive to creepy. If you use an IRB for your research (more on them in “Mind Your Research P’s and Q’s”), they’ll help you define that line.

You don’t need to have such robust data to do a real-world comparison, although obviously richer data means a better story. You can still make a case for your product using publicly available information. How do your users look different from nonusers who are pretty similar to them in terms of the behavior being changed?