Design an experiment to measure whether Kindle improved outcomes for small merchants

Problem Statement Description

Product context: Amazon is a commerce, logistics, media, devices, and cloud company; its products include Marketplace, Prime, Prime Video, Alexa devices, ads, fulfillment, and AWS.

Amazon wants to understand whether a Kindle-related product experience is improving outcomes for small merchants who use Amazon to reach customers. In this interview, “small merchants” should be treated as sellers, publishers, or creators with limited scale who may benefit from discovery, purchasing, engagement, or repeat-customer behavior driven by Kindle surfaces and Kindle customers.

Your task is to design an experiment that can credibly measure the incremental impact of Kindle on those merchants, not simply describe overall business performance. Clarify which merchant outcome matters, who is eligible, what exposure to Kindle means, how assignment would work, and how you would separate true lift from seasonality, merchant quality, catalog differences, or broader Amazon traffic changes.

This is a metrics-focused exercise: the emphasis is on defining measurable outcomes, choosing appropriate denominators and cohorts, specifying instrumentation, interpreting results, and identifying guardrails that protect customer and marketplace health.

The experience should consider:

- The exact merchant population and eligibility criteria, including size thresholds, tenure, category or content type, and baseline activity.

- What counts as Kindle exposure or treatment, and what the comparable control condition would be.

- Primary success metrics for merchant outcomes, with clear numerator, denominator, time window, and unit of analysis.

- Secondary metrics that capture discovery, conversion, repeat engagement, revenue, retention, or merchant satisfaction.

- Instrumentation needed across Kindle surfaces, Amazon purchase flows, merchant dashboards, attribution events, and customer interactions.

- Cohorts and segments to analyze, such as new versus existing merchants, high versus low catalog depth, and category-specific effects.

- Guardrail metrics for customer experience, marketplace fairness, content quality, returns/refunds, cannibalization, and Amazon economics.

- How the experiment results would be used to decide whether to expand, modify, or stop the Kindle initiative for small merchants.

The goal is to produce a clear experimental measurement plan that would help Amazon determine whether Kindle creates meaningful incremental value for small merchants while preserving a high-quality customer and marketplace experience.