Netflix Original Research: MIT Code 2023 | Netflix Technology Blog
Netflix is pleased to be the lead sponsor of the 2023 Conference on Digital Experimentation (CODE@MIT) in Cambridge, Massachusetts, for the second year in a row.This conference is a balanced blend of academic and industrial research evil smart Folks, we are proud to offer many presentations and posters at the plenary session.
Our contribution begins with a concept that is central to our understanding of A/B testing: agents!
Our first talk was by Aurelien Bibaut (with co-authors Nathan Kallus, Simon Ejdemyr and Michael Zhu), in which we discussed how to use short-term proxies to confidently measure long-term results in the presence of bias. For example, how can we estimate the impact of innovation on retention one year later without running all the experiments? We propose an estimation method that uses a cross-folding procedure and establishes valid confidence intervals until long-term effects are fully observed.
Later, Michael Zhu (with Maria Dimakopoulou, Vickie Zhang, Anh Le, and Nathan Kallus) talked about the evaluation of surrogate metric models in product decision-making. Through 200 real-life A/B tests at Netflix, we found that an alternative index model built using only 2 weeks of data resulted in the same product launch ~95% of the time compared to one based on 2 months of calls decision data. This means we can confidently run shorter tests without having to wait months for results!
Our next topic focuses on how to understand and balance competing engagement metrics; for example, should 1 hour of gaming equal 1 hour of streaming? Michael Zhu and Jordan Schafer shared a poster on how they established the Overall Evaluation Criteria (OEC) metric, which provides a holistic evaluation for A/B testing, appropriately weighing different engagement metrics to serve a single overall goal. This new framework enables fast, confident decisions in testing, and is being actively adapted as our business continues to expand into new areas.
In the second plenary session of the day, Martin Tingley takes us on a fascinating and interesting journey into complexity, exploring the key challenges in digital experimentation and how they relate to those faced by agricultural researchers a century ago How different are the challenges. He highlights different areas of complexity and provides perspective on how to address the right challenges based on business objectives.
Our final talk was by Apoorva Lal (with co-authors Samir Khan and Johan Ugander), in which we showed how to partially identify dose-response functions (DRFs) under non-parametric assumptions, providing a deeper analysis of experiments. Data outperforms standard ATE analysis. We revisit a study on algorithmic reduction of like-minded content and show how to extend binary ATE learning to answer questions about how the amount of like-minded content users see affects their political attitudes.
We had a great time connecting with the CODE@MIT community and bonded over our shared passion for not only rigorous measurements in experiments, but also statistics-themed stickers and giveaways!
We look forward to next year’s conference and hope to see you there!
Shh!We’re recruiting data scientists in a variety of fields at Netflix — check out our Open roles.
from Tech Empire Solutions https://techempiresolutions.com/netflix-original-research-mit-code-2023-netflix-technology-blog/
via https://techempiresolutions.com/
from Tech Empire Solutions https://techempiresolutions.wordpress.com/2024/02/09/netflix-original-research-mit-code-2023-netflix-technology-blog/
via https://techempiresolutions.com/
from Mary Ashley https://maryashle.wordpress.com/2024/02/09/netflix-original-research-mit-code-2023-netflix-technology-blog/
via https://techempiresolutions.com/
Comments
Post a Comment