What’s your return on ad spend?
I had never considered the possibility that ads could contribute no meaningful revenue to a company’s bottom line until Hurricane Maria blasted through the Caribbean in 2017. At the time, I was the performance marketing manager for an airline, and — in addition to causing immense human suffering — the storm crippled the tourist driven economy. The airline’s revenue forecasts shrank considerably, which meant budget cuts all around. The question posed to me: if we cut 25% of the advertising budget, what would be the commensurate revenue loss? Said another way: we pay the bills for these ads every month, what is the money actually doing?
Ad platforms claim credit for sales that would have happened regardless of ad exposure.
Our budgets were concentrated in programmatic display, social and search ads. Without doing much due diligence myself, I turned around and asked my agency partners to provide a return on ad spend (ROAS) analysis. They took the exercise seriously and provided a response based solely on our view-through attribution methodology. If we lost 25% of the budget, the agency explained, you should be prepared to lose 35% of the airline’s total quarterly revenue.
Now, this blog is anonymous and I’m not trying to out myself by being too specific, but to generalize these percentages, the agency said trimming a few million dollars from ads meant the airline could lose several hundred million dollars in revenue, which implied ROAS near 50x. This was absurd. I did not question it.
I’ll pause here to mention that before working at the airline, I cut my teeth at a giant-media-agency-holding-company working on household-name clients (a big ass bank, a big ass clothing retailer, a big ass toy company). The agency I posed this question to was part of a different giant-media-agency-holding-company, and for the most part I drank giant-media-agency-holding-company kool-aid. It had never really occurred to me that it was possible that advertising could simply be ineffective, so I was naively prepared to relay the agency’s message to my counterparts on the revenue management team forecasting the impact of the budget cuts.
Thankfully I had a director who came from an entirely different background, and understood the agency’s analysis to be preposterous. There was no way that a few million dollars cut out of paid search on Google and banner placements on Kayak was going to materially affect total revenue; that’s not how demand for air travel works (he was right). So, instead of saying that cutting budgets would cause the sky to fall (and losing our credibility), we did a cursory analysis of our own to come up with a much more realistic figure, then mapped out a plan to determine the answer to the original question.
Use holdout tests to determine the incremental impact of ads to a business’s bottom line.
To figure out the real ROI of our ads, which existed alongside robust owned and earned media activity, we implemented a series of holdout tests to determine the incremental contribution of each paid channel to total bookings and revenue. Colloquially (at least within this industry), this is referred to as ‘incrementality testing’. The result of this journey is the focus of this site. I’ll detail the methodology by channel and the results in subsequent posts (along with some of my more recent work), however suffice it to say that the result fundamentally changed my understanding of the effectiveness of advertising.
A polite summary: the effect of ad exposure on behavioral change in a large and relatively commodified vertical is inconsistent at best.
Real talk: there is an overwhelming amount of trash tier ads available for purchase which (if not outright fraudulent) do nothing to persuade someone to change their behavior.
We buy ads to generate leads and close sales (if you’re saying something about brand awareness after reading that sentence, we might have different definitions of what a brand campaign is, but also, we probably won’t be friends). There’s a ton of different ways to do this, and ~1 bajillion different companies you could work with, and even more superficial front end actions you could measure to try to explain success, but the bottom line is YOUR bottom line. Can you see a correlation between ads and sales? If not, do something different.
That sounds simple, but ‘use it or lose it’ budgeting coupled with the natural aversion giant corporations have to risk, means that marketing managers can just make their budget disappear every quarter without the burden to prove that the money spent was actually beneficial for the business.