Let’s talk about noise. Not the type your car makes that causes you to crank up the stereo, but the kind that can make your marketing testing infuriatingly difficult to read.
Most of us have probably been there – you launch a new test and the next day a news story about the brand comes out, the fed changes interest rates, or Outkast drops a reunion album. These events can disrupt otherwise steady performance and they’re totally out of our control as marketers. Even changing weather patterns can have a significant impact on certain industries.
This noise can make it very difficult for a marketer to separate true impact of an optimization from, well, everything else that’s going on in the world.
You might be reading this and thinking, “yeah, no duh, that’s why we A/B test.” That’s fair, and while my personal misgivings about Google’s Drafts & Experiments are a blog for another time, A/B testing is generally a great way to compensate for noise and generate accurate test results. There are, however, many instances in which an A/B test is simply not viable.
For example, you change bid strategy on a paid search campaign, a promotion goes live, Apple rolls out iOS 14.5. There’s no shortage of instances where keeping a holdout group for an A/B test is not possible or just not desired.
Instances like these, where a true & identical control group aren’t feasible, force marketers into pre/post analyses. Here we run into noise, are forced to report “muddy” results, or can be duped into coming to the wrong conclusions.
Causal Impact Analysis can reduce the noise and provide real statistical insight into your efforts giving you the confidence to move forward with, or revert, marketing initiatives.
What is Causal Impact Analysis?
Causal Impact is an algorithm built by Google to create a Bayesian structural time series model based on multiple control groups to estimate a series of baseline values for the time post-intervention. OK, that might be a bit intimidating but we don’t need to be statisticians to understand and use Causal impact.
Breaking it down in more digestible terms, Causal Impact:
- Identifies relationships between a test group and comparable “control” groups (i.e Campaign A consistently drives 50% as many conversions as Campaign B, 30% as many as Campaign C, etc.)
- Uses those relationships to plot expected performance post intervention or test start date. This plot is considered to be a synthetic baseline. It’s not an actual control group, but a synthetic once created with many control groups
- Compares actual results against expected results to determine the impact of the intervention
Visually, it can take a chart like this, which is difficult to draw conclusions from:
And turn it into a chart like this:
This Casual Impact visual now shows us that even though test volume is up in the post period, it has not increased as much as we would expect given the performance of other related channels.
Reading the Causal Impact Output
Causal Impact can produce a number of outputs, but two are especially useful: the graphs above and a summary of impact.
- Original: Plots the synthetic baseline created by the control groups in a dotted black line. The actual performance of the test group is plotted in solid black. The blue outline represents the bounds of a 95% confidence interval
- Pointwise: Plots the difference between the observed outcome and the predicted outcome. In the example above, the difference between actual and predicted outcomes is 0 until the intervention period. In most real world datasets there will be some variation from 0 in the pre-period. If variations in the pre-period are significant and the confidence interval very wide, either the pre period dates or control groups should be adjusted to narrow the confidence interval.
- Cumulative: Plots the cumulative difference between the observed outcome and the predicted outcome.
Summary of Impact
- Actual: Shows the average daily values & cumulative values compared to what the model predicted in the post period. The 95% CI row represents the bounds of a 95% confidence interval for the model.
- Absolute effect: Shows the average daily and cumulative difference between actual values and predicted values in the post period.
- Relative effect: Transforms the absolute effect into a percentage
A Causal Impact Use Case
Say you’re working for an eCommerce advertiser who runs Google Search, Microsoft Search, and Facebook Ads. In your Google Paid Search ads, you use a data feed to insert product prices, but you do not do this on any other channels.
Due to material cost increases, the advertiser raises prices of their products 20% across the board.
You can’t control product prices, but you can control Google Paid Search ad copy, so you turn to Causal Impact to understand how price changes have impacted click volume. Because it’s the advertiser’s busy season, clicks are up post price increase on Google Paid Search, and all other channels.
On the surface, it seems like no changes are needed. However, a savvy advertiser will want to know if less competitive prices in the ad copy are actually hurting click volume.
In this instance, you can use Google Paid Search volume as the test group with organic, direct, social, and Microsoft as control groups. Running the data through Causal Impact might show that click volume through Google Paid Search is actually lower than the model predicts.
💡 You can now recommend replacing ad copy that highlights pricing with another value proposition.
This is just one example of countless ways to leverage a Causal Impact Analysis. Other uses might include analyzing the impact of bid changes, promotions, landing page updates, and even campaign launches. The opportunities for how too apply this analysis in decision-making are virtually endless.
Limitations of Causal Impact
Like all algorithms, Causal Impact is only as good as the data it is fed.
Control groups need to be reasonably correlated to test groups and vetted before being used. If you’re running a test on a campaign group advertising "sweaters", a campaign group advertising "nachos" will probably not be a useful control.
The test and control groups should be impacted by macro events in a similar way. A campaign group advertising "scarves" would be a much better control group since the sale of sweaters and scarves are likely impacted similarly by macro events.
It’s also important to consider what’s happening in the control groups during the post period of an analysis. That scarves control group is great in theory, but if the advertiser runs out of inventory during the post period, continuing to use those campaigns as a control could produce misleading results.
Take the time to find stable and useful control groups when running a Causal Impact analysis to produce the best results.
FInally, Causal Impact is most valuable with larger data sets over a longer period of time. While it is technically possible to use this model for a one day promotion, a pre/post analysis would provide similar results in less time.
Similarly, when the model does not have enough data to work with, the confidence interval can be so wide that the analysis is of virtually no value. These caveats can rule out Causal Impact analysis for short sales of a very specific product, for instance.
Running a Causal Impact Analysis
Here’s the letdown, while you don’t need a degree in data science, running scenarios through a Causal Impact model isn’t as intuitive as a pre/post model. However, it is relatively straightforward to implement in R.
As a bonus, it won’t cost you a cent more than your time. While R might be intimidating at first, I implore you to give it time. Once you’ve successfully run your first analysis you will find it much easier and faster to run subsequent analyses.
Since this blog is not intended to be a tutorial in R, but rather to spark interest in Causal Impact, I would suggest following up with these resources:
Let us know in the comments how you are using Causal Impact or how you would like to use Causal Impact to improve performance!
Sign up for our newsletter for more posts like this delivered straight to your inbox:
