Ep. 85 Data Science: Designing an Experimentation Platform

On today’s deep dive into technical mobile marketing topics, we’re talking about how to set up an experimentation platform that puts power into the hands of product teams with smart engineering.

Today’s guest is Shan Huang, the Senior Applied Scientist at Zalando, a multinational e-commerce platform for shoes and fashion. Shan is also the co-founder of the German-Chinese Association for Artificial Intelligence, a nonprofit advancing the exchange of education, research, and public resources between Germany and China in the field of AI.

Listen on Spotify

Listen on Apple Podcasts

Listen on Google Podcasts

• Can you give me an explanation of what an experimentation platform is? What is an example of how it’s used?
• How do you set the limitations? How do you define what can be experimented?
• What are the biggest challenges to building such a platform?
• If you could go back in time and could give yourself one hint or remove one obstacle in building this platform, what would that be?
• Are you running automated optimization a/b tests?
• Are there any tricks to increase the efficiency or decrease the runtime of the experimentation?
• How do you support people knowing what experiments to run, what’s interesting, possible to test, etc?
• What was the reason for creating the experimentation platform?

• 2:53 The many use cases of Zalando’s experimentation platform
• 6:45 Putting together the right team
• 10:19 What’s important in the beginning
• 11:27 Hypothesis testing methodology
• 13:14 Adaptive experimentation
• 15:03 Methods for improving experimentation efficiency
• 18:44 Setting up a process for running a/b tests
• 23:04 Power to the product team

(6:50-6:57) “I think one of the biggest challenges is that building this kind of platform requires a team of different experts in different domains.”

(10:31-10:56) “In the beginning it’s about providing infrastructure and also helping our stakeholders with other teams learn a/b testing, understand a/b testing, because statistics is sometimes a very confusing thing--confidence interval, significance--it’s not so easy to explain. And I think it might be helpful to get a solid groundwork on this stuff.”