How to discuss experiment design in an execution interview
Answer PM A/B testing questions with a clear hypothesis, randomization unit, metric, guardrails and launch rule. Practice interpreting a worked result.
PMMockr.comQuick answer
Answer PM A/B testing questions with a clear hypothesis, randomization unit, metric, guardrails and launch rule. Practice interpreting a worked result.
Design the test before reading the result
For a proposed onboarding change, state the decision: whether the revised flow helps new accounts reach their first useful task. Define eligibility before assignment; for example, new accounts on supported app versions. Assign at account level if users can switch devices, so the same person does not see both versions. Keep control on the current experience and treatment on the specific change you want to test.
- Define the primary metric with a numerator, denominator and window: assigned eligible accounts that complete a first useful task within 24 hours divided by all assigned eligible accounts.
- Choose guardrails tied to plausible harm: later retention, crashes and support requests. Define acceptable harm before launch.
- Plan the detectable effect, baseline, variability and analysis method before selecting sample size. Duration should also cover relevant usage cycles; two weeks is not a universal rule.
- Decide how to handle positive, negative and inconclusive results. Avoid repeatedly checking a fixed-horizon test and stopping at the first favorable reading. Use a valid sequential method if early decisions are needed.
Check whether the experiment can be trusted
A sample ratio mismatch means observed assignment differs unexpectedly from the intended allocation. Investigate assignment, eligibility, logging and missing observations before interpreting impact. Microsoft Research describes this as a data-quality warning. Also check simultaneous releases and whether treatment changes who gets counted.
- If sellers serve both treatment and control buyers, discuss interference. A geographic or time-based design may fit better, with its own assumptions and analysis.
- If the result is inconclusive, explain what remains uncertain. Do not report that the feature has no effect just because a test did not establish one.
- A high conversion result does not automatically outweigh damaged retention, margin or reliability.
Practice follow-up questions about an experiment
Answer one challenge after your initial test design, then revise the recommendation if it changes your assumptions.
- Conversion rises but paid retention falls. Which result drives your decision, and why?
- How does your design change if users share accounts or belong to the same organization?
- Treatment has fewer recorded users than intended. What would you inspect before quoting an uplift?
- You cannot reach the planned sample before the launch deadline. What decision can you support with the available evidence?
Experimentation references
These primary sources explain experiment preparation, measurement quality and interpretation. The teaching case here is original and hypothetical.
Start with the decision, not just the experiment
A common mistake is jumping straight into test mechanics or metric selection without stating the business decision at hand. Interviewers want to see that you understand what the experiment is meant to inform. For example, don't say, "I'd run an A/B test," before clarifying what choice the company faces and what impact is hoped for.
Begin by articulating the core decision. For instance, "We need to decide whether to launch a new onboarding flow for our mobile app, aiming to increase user activation." This sets the stage for a purposeful experiment, rather than one driven by curiosity alone.
Frame a clear hypothesis
Your experiment should test a falsifiable statement, not just observe what happens. A weak hypothesis is, "Changing onboarding may affect retention." A stronger hypothesis is, "If we show a simplified onboarding, the percentage of users completing account setup will increase by at least 10%."
This approach clarifies what success looks like and signals to the interviewer that you understand the importance of measurable, actionable predictions. If your metric moves as expected, you have support for your decision. If not, you know the change may not be worth rolling out.
Choose a meaningful metric and explain why
Selecting the right metric is critical. Avoid choosing a vanity metric or one that's too far removed from the decision. For onboarding, the main metric might be 'activation rate,' defined as the percentage of new users who complete all onboarding steps within 24 hours.
Explain why this metric best reflects the behavior you want to influence. If activation rate increases, it’s a strong signal your new flow is more effective. If it doesn’t change, you have evidence that your update may not address the real friction.
Completing setup is an onboarding measure. For activation, define the first action that delivers product value, its time window, and the assigned-user denominator. Check whether setup completion predicts that action before treating the two as equivalent.
Worked example: Experimenting with onboarding flows
Illustrative teaching example. Suppose a productivity app tests a shorter onboarding flow. The primary metric is completion of the first useful task within 24 hours of assignment, not simply reaching the last setup screen. Assume 50,000 eligible accounts per arm, with 20,000 completing the task in control and 24,000 in treatment. These are hypothetical counts, not a claim that this sample is available within two weeks.
Control is 40% and treatment is 48%: an eight-percentage-point absolute increase and a 20% relative increase. You cannot infer the required run duration or a launch decision from these percentages alone. Check allocation, logging, uncertainty and the prespecified analysis, then compare retention and support guardrails. A result below an aspirational target can still be useful, but explain the decision rule agreed before reading the result.
If setup completion rises but first useful tasks do not, the shorter flow may only make setup easier. If first tasks rise while later retention falls, investigate the lost instruction and consider a targeted revision. Do not redefine activation after seeing which metric improved.
User segmentation: why and how
Not all users behave the same way. Segmenting users helps you uncover if the experiment’s impact varies by cohort. In the onboarding example, you might segment by device type (iOS vs. Android), acquisition channel, or geography.
Suppose you find that activation improves by 12 points for Android users, but only 5 points for iOS. This insight could influence rollout strategy or prompt further investigation into platform-specific onboarding friction.
Prioritizing and making a recommendation
After gathering results, clearly state your recommendation and rationale. In the onboarding example, even if the result is below your original target, you might recommend rollout if the benefit is material and there are no negative side effects.
Alternatively, if there is evidence of harm to other key metrics (e.g., retention, support costs), recommend a more cautious or segmented rollout, or propose a new experiment to mitigate the risk. Always tie your recommendation directly to the evidence gathered.
Addressing risks and trade-offs explicitly
Every experiment has risks, such as the possibility of introducing confusion or cannibalizing other metrics. In interviews, acknowledge these risks and describe how you would monitor them.
For the onboarding flow, a risk is that users may activate quickly but be less equipped to use the product, leading to higher churn. Propose specific ways to track these outcomes, such as monitoring 30-day retention or user feedback.
Weak reasoning versus strong justification
A weak answer: "I’d run an A/B test to see if users like the new onboarding."
A stronger answer: "I’d run an A/B test to measure if the new onboarding flow increases activation rate by at least 10 percentage points, using activation within 24 hours as the primary metric. If activation improves without harming retention or increasing support tickets, I’d recommend rollout. Otherwise, I’d reconsider or segment the rollout."
The difference is that the second answer ties the experiment to a business goal, defines success, and anticipates potential negative outcomes.
Practice exercise: timed experiment discussion
Set a timer for seven minutes. Choose a common product feature (e.g., notifications, search, profile creation). Define a business decision involving a possible change. State your hypothesis, primary metric, segmentation plan, expected outcome, and how you’d act on different results. Explain one risk and how you’d check for it.
Afterward, use this checklist: - Did I state the core decision? - Was my hypothesis clear and measurable? - Did I choose a metric that aligns with the business goal? - Did I explain segmentation and what it might reveal? - Did I address at least one risk and a way to monitor it? - Was my recommendation evidence-based?
Practice this cadence twice a week, rotating product features. Review answers critically or with a peer. Consider using PMMockr to simulate realistic prompts and get structured feedback.
FAQ
How should I answer an A/B testing interview question?
Explain the product decision, hypothesis, eligible population, assignment unit, control and treatment, primary metric, guardrails, planned analysis and decision rule. Then explain what could make the result unreliable.
Does a conversion increase mean we should launch?
Not by itself. Check measurement validity, uncertainty, practical value and guardrails. A conversion improvement can hide lower retention, margin or reliability.
How much technical detail is expected in experiment design discussions?
You should explain the logic behind your experiment, including user segmentation and metric choice, but you’re not expected to detail statistical methods or sample size calculations unless specifically asked.
What if my experiment results are inconclusive or show mixed signals?
State what additional evidence you’d seek or how you’d refine the experiment. Acknowledge uncertainty and suggest concrete next steps, such as longer observation or deeper segmentation.
Should I always propose an A/B test in execution interviews?
Not necessarily. A/B tests suit many decisions but aren’t always practical. If constraints exist (e.g., low traffic, high risk), explain alternative approaches like phased rollout, surveys, or qualitative studies, and justify your choice.