Design an experiment to measure whether Kindle improved outcomes for small merchants

Problem Statement Description

Product context: Amazon is a commerce, logistics, media, devices, and cloud company; its products include Marketplace, Prime, Prime Video, Alexa devices, ads, fulfillment, and AWS.

Amazon wants to evaluate whether a Kindle-related experience has improved outcomes for small merchants who depend on Amazon for customer reach and sales. In this context, small merchants may include independent publishers, authors, content businesses, or sellers whose products or offers are surfaced through Kindle or Kindle-adjacent shopping and discovery experiences.

The interview focuses on designing a credible experiment and measurement framework. You should clarify what “improved outcomes” means for merchants, how Kindle changes the customer or merchant workflow, and how Amazon can distinguish true merchant impact from seasonality, marketplace trends, selection effects, or changes in customer demand.

The measurement design should consider:

- The primary merchant outcome metric and its denominator, such as revenue, units sold, conversion, repeat purchase, customer acquisition, or margin per eligible merchant.

- Which merchants are eligible for the experiment, including size thresholds, category differences, tenure, geography, and prior Kindle exposure.

- The unit of randomization, such as merchant, customer, listing, geography, or traffic surface, and the trade-offs of each.

- Required instrumentation across Kindle, Amazon retail, advertising, catalog, payments, and merchant reporting systems.

- Cohorts and segmentation, including new vs. existing merchants, high- vs. low-selection merchants, and merchants with different baseline demand.

- Guardrail metrics for customer experience, Amazon economics, content quality, returns/refunds, latency, trust, and marketplace fairness.

- How long the experiment should run and what decision criteria would make the results actionable.

- Risks such as interference between treatment and control, spillover effects, inventory constraints, promotional overlap, and survivorship bias.

The goal is to define an experiment that can help Amazon decide whether Kindle is creating incremental value for small merchants, not merely shifting demand from one merchant or channel to another, and whether the impact is strong enough to justify broader product, merchandising, or platform investment.