In another post, I made the case that multi-channel sales and marketing operations should not blindly trust media reporting. For example, total sales reported by Meta may not correlate to the business’s actual sales and revenue figures. While you might think this is an issue of attribution, I would argue that marketing attribution is a tool we invented to address the higher order question of cause and effect, which is where we should instead focus our effort. So how do we structure a test to understand the degree to which advertising affects business revenue? I'll run through examples by channel at a high level in this piece, and link out to deeper thoughts on each experiment.
Randomized controlled experiments are the goal, but are not always practical.
A cursory google search can tell you that a randomized controlled experiment is the gold standard for understanding the effect of an intervention (in this case ads) on outcomes (business KPI). Since we’ve all become pharma experts post pandemic, think about it like a clinical trial. If you want to understand the effects of a new medication, you experiment by giving the treatment to a test group, and observe the effects vs. a control group (people who were given a placebo in place of the medication). It’s the same basic principle and setup in ads.
Consider a thought experiment for a moment: imagine a scenario where cookies were not being deprecated, and personal data privacy was not a general concern. Now imagine a super DMP/universal ad server existed that contained the purchase data and deterministic person to device id matching for every device on the planet. You could use this DMP to randomly split people into test and control groups and deliver media across all channels: you would have the ad exposure and conversion data of everyone. You could randomize people into test and control groups for your campaigns, show your ads to test groups, while withholding ad exposure from control groups. You could then understand if conversion rates are superior for the group that was exposed to ads vs. the group that was not exposed to ads.
This (probably to the collective relief of everyone, at least privately) does not publicly exist, but there are ways to approximate this scenario by channel.
Use an ad server to emulate a randomized controlled experiment.
I did this toward the end of the last decade - this becomes less practical as cookies continue to march toward obsoletion. Regardless, use an ad server to split devices that are about to receive an ad into test and control groups. Serve ads as normal to test groups, serve placebo ads to control groups (i.e. a smokey the bear PSA campaign). You will then have conversion data for a group that was served a placebo ad and still came to your website and purchased your product without actually seeing your ad vs. the group that was exposed to your ad.
Lift % calculation: (Test conversions rate - Control conversion rate) / control conversion rate
Ideal channels: display, online video (outside of walled gardens)
Benefits: closest thing to a randomized controlled experiment you will get on this list
Drawbacks: it’s expensive - you’re paying for the placebo ad just like you are the ad for your business. There’s DSPs that will run ghost ads / this kind of experiment, but having the ad server create the test / control groups also benefits by creating a double blind scenario (it hides the group affiliation from the DSP so they aren’t tempted to cheat the result in any way.
Use DMAs to create test / control groups for comparison (matched market test).
If you sell your product nationwide, you should be able to break your sales out by DMA, or at least zip code (which you can then roll up into DMA). Before you launch an ad campaign, find markets where sales (or revenue) by day are correlated (i.e. the sales volume by day moves in a similar pattern across both groups, aka make sure there’s good vibes from the correlation coefficient R). Split the markets into test and control groups. Geo-target the ads to the test group markets. Use the actual sales data in control markets to predict what the sales would have been in the test markets had there been no ads.
Lift % calculation: (Actual sales - Predicted sales) / Predicted sales.
Ideal channels: Any, most useful for offline channels.
Benefits: Google’s causal impact package was basically tailor made for this kind of scenario, more info on that here and here.
Drawbacks: small changes are generally hard to observe, at least with confidence. Depending on your business, and the amount of advertising you're testing, ads won’t move the needle.
More on this one here.
Create test and control groups from CRM data, audience match test group into a platform, compare conversion rate based on actual sales data.
This one is easier for perishable products, given the hard close on the sales window (I talk a lot about airlines on this dumb website). Take a list of customers, randomize it and divide it into a test group list and a control group list. Upload the test group as a custom audience in Facebook (or a DSP via Liveramp, etc.) and advertise to the test group. At the end of the campaign (and after a window to allow for whatever the purchase cycle of your product is) compare the conversion rate of the test group (total customers converted / total customers on list) to that of the control group, who were held out from ad exposure.
Lift % calculation: (Test group conversion rate - Control group conversion rate) / Control group conversion rate
Ideal channels: Social, anywhere you can target a custom audience with CRM data
Benefits: CRM lists are generally confined enough that you can leverage a reach based approach with a frequency cap and keep spend pretty low.
Drawbacks: Match rates are never 100%, and are dependent on the quality of your CRM data. May cause whoever oversees your customer data to have a heart attack.
Walled gardens will work with you to set up lift experiments, if certain conditions are met.
Social platforms like Meta and Snap will set up audience based holdout studies if your business meets certain spend minimums. Similar to the above approach, but your audience is defined at the beginning of the campaign and the platform takes care of executing the holdout. The former now requires integration with the conversion API to execute this however, (new as of 2022 ish).
Google will do something similar in search, also generally dependent on how much you spend / how useful your account reps are.
Lift % calculation: (Test group conversion rate - Control group conversion rate) / Control group conversion rate
Ideal channels: Social, search
Benefits: Low lift for brands,
Drawbacks: Lacks the double blind, but these platforms generally won’t fudge data. However, depending on your product and the channels that it can be purchased, the platform might determine there was statistically significant lift to the sales it generates, but that might not translate to total sales lift for your business. It’s possible the platform is still just cannibalizing sales that would have happened anyway through other channels.
Aaaand more on this one found here and here.
The street wants growth; incremental sales are the only thing advertising can contribute that aligns with that goal.
In the posts linked above I go into more detail about what I’ve found to be effective drivers of lift by channel, with commentary about why or why not. The key is, when you find things that are NOT incremental, cut them (banner ads; any retargeting). Do not pay for things that do not create behavioral change.
For legacy brands that employ marketing departments of hundreds of people, with 9-10 figure media budgets, this might actually be the hardest part.
Nobody wants to put entire divisions within the marketing department out of a job. Nobody actually wants to give money back to their FP&A team. If you try, that team might actually tell you no. They just want you to make the money disappear. But in the broader ‘4 P's marketing’ sense, if your ads aren’t driving incremental lift, do something else with the money. Find another way to sell things to people.