Design an experimentation dashboard for Top 10

Problem Statement Description

Product context: Netflix is a streaming entertainment company; its products include subscription video, original films and series, recommendations, profiles, games, and ad-supported plans.

Netflix’s Top 10 experience helps members quickly understand what is popular across films, series, live content, games, and localized catalogs. Imagine the team is running experiments on changes to the Top 10 module, such as ranking logic, placement, visual treatment, localization, or personalization, with particular interest in how these changes affect lapsed subscribers who return to Netflix.

Design an experimentation dashboard that helps product, data science, content, and regional teams evaluate whether a Top 10 experiment is working. The dashboard should make it easy to understand experiment health, member behavior, content discovery impact, and downstream business outcomes without over-attributing success to short-term clicks.

Your scope is not to propose the Top 10 product change itself, but to define what the dashboard should measure, how metrics should be structured, and how teams should use it to make ship, iterate, or stop decisions.

The experience should consider:

- Clear experiment context, including variant definitions, target population, geography, device, content type, and exposure rules

- Primary success metrics for Top 10 engagement and content discovery, with explicit numerators and denominators

- Lapsed-subscriber cohorts, including how “lapsed,” “returned,” and “active after return” should be defined

- Funnel instrumentation from Top 10 impression to title detail view, playback start, meaningful viewing, and continued engagement

- Guardrail metrics such as streaming quality, browsing frustration, content abandonment, churn risk, and negative impact on non-Top 10 discovery

- Segmentation by region, language, plan type, device, genre preference, and new versus returning member behavior

- Experiment validity checks, including sample size, exposure logging, randomization balance, novelty effects, and statistical confidence

- Decision usefulness for stakeholders, including how the dashboard supports launch decisions, regional rollouts, and follow-up analysis

The goal is to describe a metrics dashboard that would allow Netflix teams to evaluate Top 10 experiments rigorously, understand impact on returning lapsed subscribers, and make confident product decisions while protecting long-term member satisfaction and content discovery quality.