Design an experimentation dashboard for Wrapped at global scale

Problem Statement Description

Product context: Spotify is an audio streaming company; its products include music, podcasts, audiobooks, playlists, recommendations, creator tools, subscriptions, and ads.

Spotify Wrapped is one of Spotify’s highest-visibility annual experiences, reaching global audiences across music, podcasts, and audiobooks. As teams test new Wrapped experiences, personalization modules, sharing flows, and audiobook-specific surfaces, they need a reliable way to understand experiment performance before, during, and after launch.

Your task is to design an experimentation dashboard for Wrapped at global scale, with particular attention to audiobook listeners. The dashboard should support product, data science, engineering, marketing, and regional stakeholders who need to evaluate whether experiments improve user engagement, discovery, retention, and monetization without creating trust, quality, or reliability issues.

This is a metrics design problem, not a visual design exercise. Focus on what should be measured, how metrics should be defined, which populations and denominators matter, how experiment health should be monitored, and how the dashboard should help teams make launch, ramp, rollback, or investigation decisions.

The dashboard should consider:

- Clear primary, secondary, and guardrail metrics for Wrapped experiments, including audiobook-specific behaviors such as discovery, starts, completion, saves, purchases or consumption eligibility where relevant.

- Precise metric definitions, denominators, attribution windows, and handling of users who are ineligible, inactive, anonymous, offline, or exposed across multiple devices.

- Experiment setup visibility, including variant assignment, exposure logging, sample ratio checks, eligibility rules, market availability, platform coverage, and rollout percentage.

- Cohort and segmentation views across geography, language, subscription tier, platform, listening intensity, audiobook listeners versus non-listeners, new versus returning users, and creator/content categories.

- Instrumentation quality checks for event completeness, latency, deduplication, cross-device identity, app version differences, and potential tracking gaps during high-traffic Wrapped moments.

- Guardrails for reliability, user trust, privacy, sharing abuse, notification fatigue, ad impact, subscription impact, customer support contacts, and negative engagement signals.

- Decision usefulness for global teams, including confidence intervals, statistical power, experiment duration, seasonality around Wrapped, multiple testing risks, and criteria for ramping or stopping experiments.

The goal is to define a dashboard framework that enables Spotify teams to evaluate Wrapped experiments confidently at global scale, understand impact for audiobook listeners and broader audiences, and make timely product decisions while protecting the quality and trust of the Wrapped experience.