Ep.77 Measuring Campaign Uplift Without Device IDs

Welcome to the new Data Science segment on the Apptivate Podcast, where Johannes Haupt, Senior Data Scientist at Remerge, explores more technical topics with mobile marketers. Today’s guest shares her insights on how to measure the incremental effect of marketing campaigns without device IDs for A/B testing.

Yue Meng is the Senior Data Scientist at Delivery Hero, an on-demand delivery platform. Previously she was a marketing consultant for Ninah Consulting. She has a master’s degree in statistics. She is based in Berlin, Germany.

Listen on Spotify

Listen on Apple Podcasts

Listen on Google Podcasts

• What methodologies do you use to evaluate your various campaigns?
• What are your plans for A/B testing once device IDs are no longer available with the data privacy changes to iOS14?
• Can you give us an example of how you would analyze an offline campaign?
• How do you know if your baseline model is accurate enough?
• How would you improve your model if it wasn’t accurate enough?
• Can you use the control group only to focus on testing?

• • 1:02 Yue’s background
  • 4:45 Why Delivery Hero trusts A/B tests
  • 7:54 Plans for marketing campaign testing without device IDs
  • 11:29 KPIs for evaluating offline campaigns
  • 15:10 Measuring uplift from offline campaigns with time series
  • 21:39 Calculating accuracy with baseline models
  • 24:30 Using geo as a proxy for user-level data
  • 31:34 Looking ahead at testing solutions

(9:21-9:43) “This offline scenario is very similar to an online scenario without the device ID, I would say. So probably some models that we’re using for the offline campaigns estimation can be used on the online campaigns also. That’s what we are currently trying to do.”

(17:07-17:20) “By using the time series analysis, the simplest model you can use, you’re not adding any other factors or any other features. You only focus on the time series by itself.”

(24:30-24:41) “Now the control group on a user level is impossible. What we can try to do as a proxy is to use instead of user-level to use geo-level, for example, the city level.”