Upwave’s George London: AI Must Prove Advertising Impact

1. When AI identifies an improvement in campaign performance, what additional evidence should marketers examine before reallocating budget or changing their media strategy?

The first thing to keep in mind in any conversation about optimization is that “doing nothing” is a choice. There’s a tendency to demand too much evidence before making a change, because people assume the default plan is the right one. Our research at Upwave has found, though, that it’s very rare for the first plan to truly be the best plan, and that reallocating budget toward better-performing tactics can significantly improve brand campaign performance. That implies that the average campaign has a great deal of room for improvement from its starting point.

That said, the way to think about evidence is through the lens of statistics. Any credible analytical method should estimate not just performance but also the uncertainty around it, through a mathematically rigorous and transparent process. That means every estimate is essentially a probability distribution, a range of plausible values rather than a single number. When you compare two options, you’re comparing two probability distributions, and in the real world they will usually overlap to some degree.

In the rare case where the two distributions don’t overlap at all, that’s obviously sufficient evidence to make a highly confident change. When they overlap only partially, the rational strategy is to make a partial shift, and the less they overlap, the bigger the shift should be towards the higher performer (in reinforcement learning, this is known as “Thompson sampling”). This is the best way to benefit from the fact that some tactics probably–but-not-definitely are better than others, without overexposing yourself to the possibility that the estimated outperformance is a statistical illusion.

Of course, this also assumes that marketers have done their homework on the analytical method and data quality behind the recommendations. There’s a famous saying in computer science: “garbage in, garbage out.” If estimates are based on biased or unreliable data, then no methodology (no matter how rigorous) will be able to produce reliable results.

Even with good data, if you use oversimplified or unreliable methods that don’t properly tease apart incrementality, you can easily end up making changes that hurt your campaign performance. That can easily happen when your campaign targets people who are already likely to take the action you’re trying to drive, whether or not the ad actually influences their probability of converting. In that case, you would have gotten the same results without spending any money at all.

So do your homework on data quality and causal inference methodology, and then be willing to take a judicious amount of risk rather than demanding certainty before taking action. A useful analogy we draw is to the stock market; no individual stock is going to go up every day. But if you refuse to buy any stocks because you can’t be sure each one will be a winner, you’ll make far, far less money than if you apply a rational and prudent portfolio risk management strategy.

2. Advertising effectiveness is not limited to clicks, conversions or reach. How can AI help marketers understand changes in intent, and what evidence is needed to establish that these changes are linked to advertising?

Luckily for advertisers, there are straightforward and credible ways to measure the incremental impact of advertising on consumer intent. This is done with a method known as brand lift, where vendors like Upwave survey consumers who have been exposed to advertising and compare them to a properly constructed control group.

The math underlying brand lift (if properly done) is complex, but it does make it possible to calculate an incremental impact, i.e., the extent to which the advertising actually caused the changes in stated consumer awareness or purchase intent.

If you are working with a brand lift vendor, make sure to ask them tough questions about how their causal inference methodology works and why they are confident that they are actually measuring incremental impact (and not just identifying pre-existing differences between people who did or did not see the ad campaign).

AI is not strictly necessary for these calculations; they can be done using classical statistics and machine learning. But AI can still be extremely useful in setting up the experimental design: making sure the right survey respondents are collected and that the questionnaire is designed so it can isolate incremental changes in intent. AI can also be extremely useful for reporting campaign results in a way that makes it straightforward for stakeholders to understand and believe the performance estimates.

3. Campaign performance data often sits across disconnected platforms. How can AI help marketers connect these signals to build a complete view of advertising effectiveness without introducing inconsistencies into the analysis?

AI is very good at pulling together data from disparate sources, as long as it has proper access to the input data. At Upwave, we built our AI agent, Bayes, with properly customer-scoped access to all of that customer’s campaign performance data and campaign metadata. That lets us give customers a clear and comprehensive view of how their campaign is performing, both in isolation and against comparable campaigns, their own and (anonymized) other advertisers’.

APIs and MCP connectors are making it increasingly straightforward for marketers to pull together data from multiple vendors and bring it into one place where the AI can analyze it together. As long as each data source is adequately transparent about how its data was generated and the methodology behind any upstream analytics, the AI can understand the caveats and apply them appropriately to stitch together the best understanding the data allows, given its inherent limitations. For example, if two different platforms use different control methodologies, the AI can dig deep into the reliability of these respective approaches. It can warn the user if one estimate should be considered less trustworthy than another, perhaps leading the advertiser to apply a wider range of uncertainty than what the estimating platform reports by default.

4. AI can identify patterns in large datasets, but a pattern does not explain what caused an outcome. How should marketers distinguish between an AI correlation and evidence that an advertising campaign drove a change in brand performance?

Luckily, there is an entire academic discipline known as “causal inference” dedicated to answering exactly this question: telling when one thing caused another versus when two things are merely correlated.

Unfortunately, the math behind causal inference is not trivial, and it takes real planning up front to make sure you measure the right variables to isolate causal impact.

Upwave has spent years building a rigorous and reliable causal inference pipeline. It is at the core of all the measurement we do. It works by identifying features that are correlated with both the probability of exposure to the ad and a user’s opinion of the brand (for example, when a campaign is retargeted toward past purchasers of a brand). We can use this information to “deconfound” the estimates and make sure that, for example, we are comparing the impact of the ads on consideration among past purchasers who saw the ad versus among past purchasers who did not, when building up to the final total estimate. We are proud of this method, and happy to discuss it at great depth with any interested advertisers.

Because of the rigor of our methods, when advertisers use Upwave, they can be confident that we are reporting the true incremental impact of their advertising campaigns.

Advertisers who care about incrementality (which should be every advertiser!) should make sure to ask their vendors tough questions about what causal inference techniques they use and how they know they are measuring causal impact correctly. Ask, for example, how their methods measure and adjust for confounding in their impact estimates, and how they know they’re capturing and applying the right set of confounding variables.

Advertisers should only work with measurement vendors who are transparent about their methods and can clearly and convincingly explain how they apply causal inference techniques.

5. Advertising effectiveness is influenced by factors beyond the campaign itself, including competitor activity, pricing, distribution and market conditions. How should AI measurement account for these external variables before recommending changes to media investment?

When it comes to measuring the impact of advertising, advertisers need to think carefully about the question they are actually trying to answer. At Upwave, we answer the narrow counterfactual question: what would have happened to perceptions of this brand if this specific campaign had never run, and everything else about the world had stayed the same?

Variables like competitor activity and pricing do strongly influence overall, population-level purchase intent. This is why it’s really important to frame the question correctly and to use rigorous, mathematically sound causal inference techniques that isolate the true counterfactual impact.

It’s also why it’s not a good idea to use population-level surveys, such as brand trackers, to estimate the impact of advertising. It’s very possible for consideration, favorability, or purchase intent, to plummet on a brand tracker even while an advertiser is running a very successful campaign, for example when the company is going through a scandal. In that case, purchase intent would have plummeted even faster without the campaign!

When planning campaigns and thinking about the impact advertising will have going forward, advertisers need to be realistic about what it can and cannot do. A good ad campaign can make purchase intent and favorability higher among the people it reaches than it would have been without the campaign.

But simple arithmetic suggests that campaigns need to be extremely large to move population-level KPIs. The US has 340 million people, so an ad campaign would need to make ~3 million people newly aware of a brand just to increase population-level brand awareness by 1 percentage point. Even the best ad campaigns will tend to have smaller impacts than macroeconomic factors such as changes in consumers’ disposable income.

It’s still a good idea, though, to run effective advertising when measurement results suggest that an advertiser can do so profitably.

Although AI is not directly necessary to measure the counterfactual impact of advertising, it can be extremely useful to helps advertisers contextualize and interpret the results from brand lift measurement. It can also help them assemble more complete predictive models of how the many variables under their control (such as pricing, promotions, press, and advertising effectiveness) can be expected to interact to drive long-term sales. Upwave has some very exciting research in the labs that’s working to quantify the direct dollar impact of brand advertising on sales, and we look forward to being able to share our findings soon.

6. Many marketing teams already have dashboards that explain what happened during a campaign. What separates an AI performance report from actionable intelligence that helps a CMO decide what to do next?

At Upwave, we believe data is only as useful as the decisions it helps you make.

Brand lift results have traditionally been reported in 60+ slide decks that people flip through rather than read in depth, and then archive. There is value in using AI to produce those decks in minutes instead of having human analysts work on them for weeks. But the real value comes from distilling an overwhelming number of data points down into the key takeaways that meaningfully change strategy.

This does require CMOs to think about what’s actually flexible in their media plans and what’s not. If an advertiser has large upfront commitments to a TV network, learning that the network is underperforming may not have much impact in the short term. But if the advertiser is running four different creatives and learns that one of them is outperforming, they can shift budget toward that winning creative.

On a recent campaign we measured, an advertiser identified that ads served to certain audiences were outperforming, and by shifting more delivery to those audiences were able to increase lift by 30%.

The great power of AI is that it can take many complicated constraints into account, search the haystack of data for the needles of useful insight, and present recommendations that line up directly with what’s useful and actionable for the CMO.

At Upwave, we are building our AI analyst, Bayes, to learn and remember the context and learning agendas advertisers give it, so it can automatically customize its recommendations to what each advertiser will find most useful.

7. Which parts of brand measurement can AI handle with limited human intervention, and where should experience analysts retain responsibility for interpreting results and making recommendations?

There’s a long-standing saying in AI that a computer should never make a decision, because it can never be held accountable. This will remain true for the foreseeable future.

The operational capabilities of AI agents are improving at a breakneck pace, and the days of human analysts crunching numbers and formatting PowerPoint slides are rapidly coming to a close. Upwave’s AI is already extremely useful for interpreting results, identifying opportunities for optimization, constructing deliverables like slides and charts, and finding the cleanest, clearest words to communicate findings to stakeholders. That said, even when AI makes a compelling recommendation, it still cannot decide whether that recommendation is actually actionable and advantageous for the advertiser’s business, or whether it realistically accounts for all of the advertiser’s business constraints and objectives.

AI is very good at working with the information it’s given, but it doesn’t know what it doesn’t know. Humans will need to remain the ultimate consumers of the analysis and insights the AI produces, and they will need to be the ones who make the final decisions and take final responsibility for actions taken and money moved based on its recommendations.

8. Looking ahead to 2027, what is the biggest mistake you expect marketing leaders to make when adopting AI for brand measurement, and what should they prioritize instead?

I think the biggest mistake marketers are likely to make with AI is adopting it too slowly. AI capabilities are improving at an incredible rate, and the accuracy and reliability of today’s AI agents are miles ahead of where they were even one year ago. People who base their opinions about the limits of AI on the early releases of ChatGPT will substantially underestimate the value and leverage that modern AI can provide.

Organizations still need to retain ultimate accountability for the results AI produces, so they have to apply appropriate scrutiny to the AI’s work to make sure it is correct and complete. They may also need to make organizational changes to make sure that the AI has access to the data and tools it needs to move fast, and isn’t being slowed down by “waiting for humans.”

That said, organizations that lean cautiously but heavily into AI for brand measurement will be able to make much faster and better decisions about how to maximize the return on their advertising dollars. Businesses that can advertise more efficiently will grow faster, and they will be the ones that win in competitive markets.

Marketers should always be prudent and thoughtful about adopting new technology, but the ones who let skepticism or cynicism dominate will be left behind.

About George London, CTO, Upwave

George is a seasoned technology leader who has spent his whole career helping companies use data to make better decisions. George started his career doing macroeconomic modeling and investment research at Bridgewater Associates (the world’s largest hedge fund), and then founded a startup that used data to help consumers explore and discover music.

About Upwave 

Upwave is the Brand Outcomes Measurement Platform. The only company entirely focused on measuring and optimizing brand lift driven by advertising, Upwave has measured Brand Outcomes across more than 3,000 brands and is trusted by the world’s leading advertisers, agencies, and media partners. Upwave brings science to top-of-funnel with its robust, AI-driven SaaS platform, providing real-time measurement across CTV, Digital, Retail Media, Social, Streaming Audio, Linear, and Addressable. The company is based in San Francisco and New York and backed by leading Silicon Valley venture capital investors.