Why Your CX Initiatives Need Controlled Tests to Prove Value

Do your customer experience (CX) initiatives deliver results? And do those results resonate with your company’s decision makers?

The burden of proof lies with CX teams. Business leaders demand clear evidence that CX investments are worth the cost, and this is especially true at the 80% of businesses where CX isn’t central to their brand identity. Unfortunately, this can often become a stumbling block. According to a Forrester study more than half of CX professionals report being unable to demonstrate the ROI of their CX projects. Without the ability to measure the effectiveness of a CX initiative, justifying its existence becomes incredibly difficult—if not impossible.

But many CX professionals are neglecting an effective tool for demonstrating business value: controlled tests. Running a study using a control group is a powerful way to prove that CX initiatives influence key business metrics—and it’s simpler than it might seem. Keep reading to learn the advantages of controlled tests and eight steps to run them successfully.

What Is a Controlled Test?

A controlled test is a method for evaluating the effectiveness of a treatment or intervention. It’s usually done by controlling all the variables that could influence an outcome, and then changing only one variable at a time. The difference in outcome, then, is evidence of the treatment’s benefit.

Controlled tests often take the form of a study using a treated group and an untreated group—a.k.a. the control group.

You’re probably familiar with this concept, as this is what’s used in medical trials. Suppose you get 100 people in one building for a month and feed everyone the same diet, except you give one half of the participants a diet drug and give the other half a sugar pill. and the difference in what you see in metrics is the benefit of the diet drug.

Well, this method works for business KPIs, too! In a CX controlled test, control group participants don’t receive the program being evaluated—such as appointment reminders, self-service tools or proactive fraud alerts. Then you’re measuring the lift in KPIs and attributing it to your program.

Why Do You Need a Controlled Test for CX Measurement?

A controlled test gives you a baseline for comparison against the group you treat with the intervention—e.g., a new CX solution. This helps you determine the effectiveness of the new treatment by isolating its effects from other variables. If participants in an experimental group have a better outcome than control group members, that’s persuasive evidence that the intervention was effective.

Why is this important? Think of a specific CX KPI you’re tracking, like Net Promoter Score (NPS) or inbound contact center volume, and then the myriad factors that can influence it. Things outside your control, like pricing or rate changes or even macroeconomic factors, can affect those KPIs independently of your CX intervention. Without a control group, you can’t effectively isolate your CX treatment from those other variables to show its real, convincing impact.

A control group helps you…

Demonstrate ROI

ROI calculation requires two pieces of information:

• What did you invest?
• What did it return?

To decide whether a CX initiative is worth doing (or continuing), you must determine whether the results (return) justify the investment in staffing, technology and upfront costs to fund the initiative. But before you can calculate the returns, you must first measure the impact of the initiative. When you use a control group, the difference in any business KPI (sales conversions, churn rate, customer care contact rate and handle time, etc.) between the intervention and control groups is your quantifiable return.

Isolate the impact of simultaneous initiatives

Businesses often have several initiatives going on at once, making it difficult to pinpoint which program was responsible for changes. Every team claims the credit for positive outcomes. If experimental and control group participants have access to the same promotions and programs (except the initiative you’re evaluating), the difference in results is attributable to your CX program.

Avoid the limitations of before-and-after time series analysis

Many business leaders use this approach to compare trends in metrics like sales conversions or churn rates before and after an intervention. However, this method is often unsuitable—especially in recurring relationship businesses like subscription models— due to seasonal and event-driven fluctuations. Numerous external factors influence customer behavior and business outcomes, including:

• Changing market conditions
• Natural disasters such as hurricanes
• National emergencies such as the COVID-19 pandemic
• Competitor promotions
• Price changes
• And a whole lot more

Using a control group allows you to isolate the impact of the treatment from these situational factors. Assuming people in the experimental and control groups are exposed to these situational factors at similar rates, you can attribute the different outcomes to your CX initiative.

How to Set up a CX Controlled Test

Follow these eight steps to set up a controlled CX experiment:

1. Clearly define what action you’ll take. For example, provide a digital bill explanation tool to help customers understand changes in their monthly wireless bills.
2. Indicate what business metrics you expect the action to influence. In this case, implementing the solution will decrease contact rates and increase average revenue per user (ARPU).
3. Determine how much change you need to see in the metrics to break even on the intervention, or profit from it. Keep in mind, too, that you’ll need to be able to optimize once you see the results.
4. Set the study time period. For how many weeks or months will you need to run the experiment to get a reliable sample? It depends on the use case. For the digital bill explanation example, you might need to cover several billing cycles.
5. Establish your test sample. There are two main elements to consider here:
   1. Size: How large should the control group be so you can have confidence that the results are representative? The larger you can make the group, the quicker you can see business impact that helps you make decisions.
   2. Segment: While it’s ideal to identify a representative population of customers that you can break into groups, this can be a complex task for most CX organizations. We’ve found that using a random group still produces good data and is much more manageable.
6. Review the data for results. Then optimize the solution so it produces the best results. For example, what adjustments could be made to the bill explanation tool’s delivery that might further decrease contact rates?
7. Calculate the difference in outcomes (metrics) between the two groups. Let’s say the treated group participants contacted at a 30% rate during the 8-week study period, while the control group contacted at a rate of 20%. This indicates an overall 33% reduction in inbound contact center volume.
8. Calculate the monetary value of the change in KPI: That’s the real financial impact attributable to your bill explanation solution. Take that 33% contact rate reduction and apply it across your monthly contact volume and the savings per avoided contact, and you can identify the direct savings. To calculate ROI, divide the cost of the initiative by the savings—now you have your ROI percentage.

Prove the Impact of Your CX Initiatives—and Secure Future Investments

You can show that your CX initiatives—not external factors—are responsible for improvements in KPIs that matter to decision makers. It takes a little scientific rigor, running CX measurement experiments that prove the value of the project—helping your team secure funding for more.