Design an experiment to measure whether Messenger improved outcomes for families

Problem Statement Description

Product context: Meta is a social technology company; its products include Facebook, Instagram, WhatsApp, Messenger, Threads, Quest, creator tools, and ads.

Meta wants to understand whether changes to Messenger are meaningfully improving outcomes for families who use the product to stay connected, coordinate daily life, and maintain relationships across households, generations, and distances. In this interview, you are asked to design an experiment that can measure whether Messenger is creating better family outcomes, not just increasing generic engagement.

Assume the relevant users may include parents, children, grandparents, siblings, extended relatives, and family groups who communicate through 1:1 chats, group chats, calls, media sharing, reactions, reminders, or other Messenger surfaces. The challenge is to define what “improved outcomes for families” means in measurable product terms while respecting privacy, safety, and the fact that family value may be qualitative, long-term, and unevenly distributed across members.

Your task is to propose an experimental approach that could help Meta make a decision about whether a Messenger experience, feature, or product change is beneficial for family users. The focus should be on metric design, experiment structure, measurement validity, and interpretation of results rather than on designing the feature itself.

The measurement plan should consider:

- How to define the target family user or family group cohort, including the denominator for any primary metric.

- What primary outcome metric best captures improved family connection, coordination, or relationship value.

- What supporting metrics help explain behavior across messaging, calls, group activity, media sharing, or repeat use.

- How instrumentation would identify relevant family interactions without relying on invasive or unsafe assumptions.

- How to segment results by family type, geography, distance between members, group size, new versus existing users, or communication modality.

- What guardrail metrics are needed for privacy, safety, spam, unwanted contact, notification fatigue, and overall Messenger health.

- How long the experiment should run to detect meaningful family behavior changes and avoid misleading short-term engagement spikes.

- How the results would inform a launch, iteration, or rollback decision.

The goal is to demonstrate that you can translate an ambiguous social outcome into a rigorous, decision-useful experiment for a large-scale messaging product, balancing measurable impact with user trust, family safety, and Meta’s responsibility to evaluate product changes carefully.