Design an experiment to measure whether Logistics improved outcomes for small merchants

Problem Statement Description

Product context: Amazon is a commerce, logistics, media, devices, and cloud company; its products include Marketplace, Prime, Prime Video, Alexa devices, ads, fulfillment, and AWS.

Amazon wants to understand whether its Logistics offering is meaningfully improving outcomes for small merchants who sell through or alongside Amazon’s commerce ecosystem. These merchants often face constraints around fulfillment speed, delivery reliability, inventory placement, shipping costs, returns handling, and customer trust, all of which can affect sales growth and retention.

You are asked to design an experiment that can credibly measure the impact of Logistics on this merchant segment. The focus is not just whether delivery performance improves, but whether those operational improvements translate into better merchant outcomes and a healthier customer experience.

Your response should define what “improved outcomes” means, how the experiment would be structured, what metrics should be tracked, and how Amazon should interpret the results. Be explicit about the population, treatment and control groups, measurement window, instrumentation, and trade-offs between merchant benefit, customer experience, and operational cost.

The experience should consider:

- Which small merchants are eligible for the experiment and how “small merchant” should be defined.

- The primary success metric, including numerator, denominator, and why it reflects merchant improvement.

- Supporting metrics across delivery speed, delivery promise accuracy, order volume, conversion, retention, returns, and merchant satisfaction.

- Guardrail metrics for customer experience, logistics cost, late deliveries, cancellations, defects, and marketplace fairness.

- How to randomize or otherwise create a credible comparison group while avoiding selection bias.

- How to instrument merchant, order, shipment, and customer-level events across the funnel.

- Which cohorts should be analyzed separately, such as category, region, merchant maturity, inventory profile, or baseline fulfillment performance.

- How long the experiment should run and what decision the results should support.

The goal is to evaluate whether the candidate can design a rigorous, decision-useful metrics experiment in an Amazon Logistics context, balancing statistical validity with operational realities and the needs of small merchants.