Conversion in Amazon Shopping fell after a redesign. How would you investigate

Problem Statement Description

Product context: Amazon is a commerce, logistics, media, devices, and cloud company; its products include Marketplace, Prime, Prime Video, Alexa devices, ads, fulfillment, and AWS.

Amazon Shopping has recently shipped a redesign to part of the customer shopping experience, and overall conversion has fallen afterward. You are asked to investigate the decline as a product manager, with attention to how the change may have affected customers discovering, evaluating, and purchasing products, including items sold by small merchants on the marketplace.

This is a root-cause analysis question. Your task is not to propose a new redesign immediately, but to structure how you would determine whether the conversion drop is real, where in the funnel it occurred, which users or merchant segments are affected, and whether the redesign is the likely cause versus other marketplace, traffic, pricing, inventory, or measurement factors.

The investigation should reflect Amazon’s environment: high-volume commerce, multiple surfaces and devices, personalized recommendations, search and browse behavior, cart and checkout flows, marketplace sellers, logistics promises, and operational systems that can independently affect purchase completion.

The experience should consider:

- How you would define the conversion metric, including numerator, denominator, time window, and whether it refers to session, visitor, product-view, cart, or checkout conversion.

- How you would confirm the anomaly is statistically and operationally real, including seasonality, traffic mix, experiment exposure, and instrumentation changes.

- How you would segment the decline by platform, geography, customer cohort, acquisition channel, product category, merchant type, Prime status, and new versus returning users.

- How you would inspect the shopping funnel to locate the biggest drop-off points, such as search results, product detail pages, add-to-cart, cart review, checkout, payment, or delivery selection.

- What hypotheses you would test around the redesign, including usability friction, reduced trust signals, changed ranking or merchandising, latency, missing information, pricing visibility, delivery promise clarity, or small-merchant offer presentation.

- What evidence you would seek from logs, A/B test data, analytics dashboards, customer feedback, seller signals, operational metrics, and support contacts.

- How you would think about immediate mitigation, controlled rollback, further experimentation, and communication with design, engineering, analytics, operations, and marketplace teams.

- What prevention mechanisms you would recommend for future redesigns, such as guardrail metrics, phased rollout, monitoring, alerting, and pre-launch validation.

Your goal is to demonstrate a clear, disciplined RCA approach that separates symptom from cause, protects customer trust and marketplace health, and leads to a confident decision on whether to rollback, iterate, or continue the redesign with targeted fixes.