google is the digital real estate developer you never asked for

Settle in, I think this is the longest thing I wrote. Apologies in advance.

As mentioned in Lift 101, you can use DMAs to create test and control groups.

An airline is a good use case for this type of solution. You have markets that are served by the airline, which line up pretty well with DMAs as defined by Google. When you serve ads to a specific DMA, you can track clicks from those ads to flight searches originating at the corresponding airport(s). There’s some leakage for ads served in one DMA occurring in searches to airports across the country, but that’s the long tail outcome and doesn’t skew the result.

So to measure search lift, we randomized DMAs into test and control groups and bought search ads in the test group DMAs, and kept the control DMAs dark.

*When we structured a lift test for search media, ~80% of the search budget ended up on Google (vs. Bing or Yahoo), so we focused on that partner. Does Google get mad about their dumb brand image being on random websites?

Use the control group to infer what would have happened in the test markets had there been no search ads.

The KPI of interest in the test and control DMAs (sales by day, revenue, website visits, etc.) should be correlated before the test goes live. You can use a package like Google’s causal impact to run this if you’re comfortable with a little coding, or just create the counterfactual using linear regression tools in excel.

Sample counterfactual measurement:

The black line represents our control group KPI, let’s say it’s sales by day. The blue line is test group sales by day. The vertical dotted line represents the start of the test, where ads are turned on in test markets. The dotted blue line is the counterfactual, or what we can infer would have happened in the test markets if ads were absent, based on what’s happening in the control market (this is why you want to check the correlation between KPIs in the markets before the test). As noted in the chart, there’s roughly a 1.2x difference by day between the control and the test group, which is what is used to calculate the counterfactual. The difference between the counterfactual and the actual sales in the test group is the lift.

In a perfect world (like the example above), you would see clear lift in the test group that you could attribute to ads.

Overall, we measured statistically significant lift to website traffic and bookings completed on the site; however there was no lift to total revenue in the test markets.

This point can be a bit nuanced, especially if your brand is like an airline and doesn’t have complete control over the point of sale. Ads tend to work: they educate people about things they didn’t know about, and digital ads will drive people to landing pages they may not have visited otherwise, which is what the test demonstrated. However, since the airline also sold tickets via online travel agencies such as Expedia, the lift created by search ads was only happening in the direct sales channel (on the airline’s dotcom). In the control markets (where there were no search ads), the airline sold the same number of tickets it would have without paid search ads, but the share of transactions happening through third parties was greater.

Always evaluate the lift based on total business metrics, from 1st party sources, especially when your business has multiple points of sale. Google's story about search generating incremental lift wasn't entirely inaccurate, it was just incomplete.

Brand search ads are extortion, and you probably need to pay some kind of tribute.

The result of the test outlined above determined that brand search ads are almost entirely non-incremental. Again, just thinking about who this customer is and what they are looking for, the majority of the traffic is derived from people who entered the airline’s name (or an approximation thereof) into Google, and want to buy a flight. It isn’t a surprise that the traffic here would have bought a ticket from the airline regardless of clicking on a paid search ad.

So why pay for brand search? I’m going to keep the lecture on auction dynamics here at a high level, but this is where Google becomes the Digital Real Estate Developer you never asked for: your competitors can pay to show up in the results when people search for YOUR brand. This can result in losing traffic and sales, which is worth the budget to stave off,  if only to avoid getting curt emails with sloppy screenshots from your executive team.

Honestly though, depending on your brand and the aggressiveness of your competitors you might be able to sit back and let organic search work for you.

There’s a more nefarious element to this as well. Back in the day, this airline did not charge customers fees to change or cancel a flight, while most other airlines did. Sometime after we backed off our branded search presence, a former cop who worked in our corporate security office wandered up to my desk. A scam service started showing up in the branded search results, not explicitly representing themselves as the airline, but not trying to dissuade anyone of that notion. They’d take people’s money then call the airline impersonating the customer and claim bereavement to offset ticket price changes. It wasn’t exactly killing the bottom line, but it was a headache for customer service. You can easily push these kinds of actors off your corner just by putting some cash to the Google machine.

Non-brand search ads have a better shot at generating incremental sales but will never be efficient.

These are the people who are the closest to buying something but are free agents. Sticking with our airline example here, these people are searching for ‘flights to X-destination’ and comparing options instead of going to airline.com and looking at the a singular airline's flight options. This is where the incremental site traffic and direct sales came from in our experiment, but the cost of this traffic was astronomical. Even if 100% of the bookings were net incremental, the revenue would never have offset the cost of the clicks.

REMEMBER: the primary objective of the ad platforms you use is to spend your entire budget.

NEXT UP: JUST GIVE YOUR MONEY TO META AND WORRY ABOUT OTHER SOCIAL NETWORKS LATER