Design an experiment to measure whether Messenger improved outcomes for families

Problem Statement Description

Product context: Meta is a social technology company; its products include Facebook, Instagram, WhatsApp, Messenger, Threads, Quest, creator tools, and ads.

Meta wants to understand whether Messenger is genuinely improving outcomes for families, not just increasing message volume. In this context, “families” may include parents, children, siblings, grandparents, caregivers, and extended relatives who use Messenger to coordinate, stay emotionally connected, share media, or support each other across households and distance.

Your task is to design an experiment that can credibly measure whether Messenger creates better family outcomes. This is a hard metrics problem because the desired impact may be emotional, relational, safety-sensitive, and long-term, while the measurable product behaviors may be indirect proxies such as communication frequency, responsiveness, group activity, media sharing, calls, or coordination behaviors.

Assume the product team may be evaluating a Messenger experience, feature, or set of interventions intended to help families connect more meaningfully. You should define what “improved outcomes for families” means, how to measure it, how to structure the experiment, and how to interpret results without over-optimizing for shallow engagement.

The experience should consider:

- The primary success metric, including its numerator, denominator, time window, and why it reflects a meaningful family outcome.

- How to identify or infer family cohorts while respecting privacy, safety, and data minimization expectations.

- Experiment design choices, including treatment/control assignment, unit of randomization, duration, holdouts, and network effects within family groups.

- Instrumentation needed to capture relevant behaviors such as messaging, calling, group coordination, media sharing, response patterns, and retention.

- Cohort cuts such as long-distance families, multi-generational groups, parents with children, new family groups, high-activity vs. low-activity users, and geography or age-based constraints.

- Guardrail metrics for spam, unwanted contact, notification fatigue, privacy complaints, safety reports, blocking, muting, and negative sentiment.

- How to distinguish durable family value from short-term engagement spikes or novelty effects.

- How the experiment results would support a product decision, including launch, iteration, rollback, or further research.

The goal is to frame a rigorous measurement approach that helps Meta decide whether Messenger is improving the quality and usefulness of family communication in a way that is measurable, safe, privacy-conscious, and actionable for product decision-making.