A compilation of Incrementality Resources - Scientific Papers

We’ve been using Incrementality measurements in App Retargeting for three years now and are proud to be able to offer all clients cost-free continuous uplift tests that provide transparency on the true value of campaigns. It took a lot of research to get to this point, and we’ve been gradually compiling a library of useful resources. We thought we’d share some of the key documents that we’ve been referring to internally while developing our custom approach. Happy reading!

The ins and outs of incrementality testing

Samuel Harries, Adjust

Incrementality is the Best Way to Prove Your Advertising is Working. Here’s How to Measure It

Natraj Ramachandran, Media Math

A Revolution in Measuring Ad Effectiveness: Knowing Who Would Have Been Exposed

Think With Google

Ghost Ads: Improving the Economics of Measuring Online Ad Effectiveness

Garrett A. Johnson, Randall A. Lewis, and Elmar I. Nubbemeyer Journal of Marketing Research, Volume: 54 issue: 6, page(s): 867-884

Abstract: “To measure the effects of advertising, marketers must know how consumers would behave had they not seen the ads. The authors develop a methodology they call “ghost ads,” which facilitates this comparison by identifying the control group counterparts of the exposed consumers in a randomized experiment. The authors show that, relative to public service announcement and intent-to-treat A/B tests, ghost ads can reduce the cost of experimentation, improve measurement precision, deliver the relevant strategic baseline, and work with modern ad platforms that optimize ad delivery in real time. The authors also describe a variant, “predicted ghost ad” methodology, which is compatible with online display advertising platforms; their implementation records more than 100 million predicted ghost ads per day. The authors demonstrate the methodology with an online retailer’s display retargeting campaign. They show novel evidence that retargeting can work: the ads lifted website visits by 17.2% and purchases by 10.5%. Compared with intent-to-treat and public service announcement experiments, advertisers can measure ad lift just as precisely while spending at least an order of magnitude less.”

The Online Display Ad Effectiveness Funnel & Carryover: A Meta-study of Predicted Ghost Ad Experiments

Garrett A. Johnson, Randall A. Lewis & Elmar I. Nubbemeyer

Abstract: “We present a meta-study of 432 online display ad field experiments on the Google Display Network. The experiments feature 431 advertisers from varied industries and on average include 4 million users and last 20 days. These experiments employ the Predicted Ghost Ad experimentation methodology. Across experiments, we find median lifts of 16% in site visits and 8% in conversions. We relate the baseline attrition as consumers move down the purchase process—the marketing funnel—to the incremental effect of ads on the consumer purchase process—the ‘ad effectiveness funnel.’ We find that incremental site visitors are less likely to convert than baseline visitors: a 10% lift in site visitors translates into a 5-7% lift in converters. We then examine the carryover effect of the campaigns to determine whether the in-campaign lift carries forward after the campaign or instead only causes users to take an action earlier than they otherwise would have. We find that most campaigns have a modest, positive carryover four weeks after the campaign ended with a further 6% lift in visitors and 16% lift in visits on average, relative to the in-campaign lift”

CONSUMER HETEROGENEITY AND PAID SEARCH EFFECTIVENESS: A LARGE SCALE FIELD EXPERIMENT

Tom Blake, Chris Nosko, Steven Tadelis

Abstract: “Internet advertising has been the fastest growing advertising channel in recent years with paid search ads comprising the bulk of this revenue. We present results from a series of large scale field experiments done at eBay that were designed to measure the causal effectiveness of paid search ads. Because search clicks and purchase behavior are correlated, we show that returns from paid search are a fraction of conventional non-experimental estimates. As an extreme case, we show that brand-keyword ads have no measurable short-term benefits. For non-brand keywords we find that new and infrequent users are positively influenced by ads but that more frequent users whose purchasing behavior is not influenced by ads account for most of the advertising expenses, resulting in average returns that are negative.”

Uplift Modeling with ROC: An SRL Case Study

Houssam Nassif, Finn Kuusisto, Elizabeth S. Burnside, and Jude Shavlik University of Wisconsin, Madison, USA

Abstract: “Uplift modeling is a classification method that determines the incremental impact of an action on a given population. Uplift modeling aims at maximizing the area under the uplift curve, which is the difference between the subject and control sets’ area under the lift curve. Lift and uplift curves are seldom used outside of the marketing domain, whereas the related ROC curve is frequently used in multiple areas. Achieving a good uplift using an ROC-based model instead of lift may be more intuitive in several areas, and may help uplift modeling reach a wider audience. We alter SAYL, an uplift-modeling statistical relational learner, to use ROC instead of lift. We test our approach on a screening mammography dataset. SAYL-ROC outperforms SAYL on our data, though not significantly, suggesting that ROC can be used for uplift modeling. On the other hand, SAYL-ROC returns larger models, reducing interpretability.”

Incrementality Bidding & Attribution

Randall Lewis, Jeffrey Wong, Netflix

Abstract: “The causal effect of showing an ad to a potential customer versus not, commonly referred to as “incrementality,” is the fundamental question of advertising effectiveness. In digital advertising three major puzzle pieces are central to rigorously quantifying advertising incrementality: ad buying/bidding/pricing, attribution, and experimentation. Building on the foundations of machine learning and causal econometrics, we propose a methodology that unifies these three concepts into a computationally viable model of both bidding and attribution which spans the randomization, training, cross validation, scoring, and conversion attribution of advertising’s causal effects. Implementation of this approach is likely to secure a significant improvement in the return on investment of advertising.”

Goodbye last-click attribution, hello incremental revenue

Ryan Hofmann