Conversion in AWS fell after a redesign. How would you investigate

Problem Statement Description

Product context: Amazon is a commerce, logistics, media, devices, and cloud company; its products include Marketplace, Prime, Prime Video, Alexa devices, ads, fulfillment, and AWS. AWS is Amazon's cloud platform; its products include compute, storage, databases, analytics, networking, security, machine learning, and developer tools.

You are a product manager on AWS responsible for the acquisition and onboarding experience for small merchants evaluating cloud services for their commerce operations. AWS recently launched a redesign of a key conversion flow, such as the marketing landing page, pricing comparison path, account creation funnel, or “get started” onboarding journey. After the redesign, the team observes a meaningful drop in conversion.

Your task is to investigate the decline in a structured way. Treat this as a root-cause analysis problem: clarify what conversion means, confirm whether the drop is real, isolate where in the funnel it occurred, and develop hypotheses about whether the redesign, measurement, traffic mix, external factors, or operational issues caused the change.

The investigation should account for AWS’s context: a technical product with multiple customer segments, high trust and reliability expectations, complex pricing, and small merchants who may be less cloud-savvy than enterprise buyers. The answer should focus on how you would diagnose the issue, what evidence you would seek, and how you would decide what to do next.

The experience should consider:

- The exact conversion metric, denominator, funnel step, time window, and baseline being compared

- Whether the anomaly is statistically significant and consistent across devices, geographies, traffic sources, customer types, and merchant sizes

- Instrumentation and tracking checks, including event definition changes, broken tags, attribution changes, or data pipeline delays after the redesign

- Funnel segmentation to identify where users are dropping off: landing page, pricing, documentation, sign-up, payment, identity verification, or first service activation

- Product hypotheses related to the redesign, such as unclear value proposition, increased cognitive load, pricing confusion, trust concerns, performance issues, or accessibility problems

- External and operational factors, including campaign mix, seasonality, competitor activity, AWS service incidents, policy changes, or sales/support changes

- Evidence-gathering methods such as logs, analytics, session replays, user research, customer support tickets, A/B results, and cohort analysis

- Immediate mitigation, longer-term prevention, and how to communicate findings to engineering, design, marketing, analytics, and leadership

The goal is to demonstrate a clear, data-driven RCA approach that separates correlation from causation, narrows the problem quickly, protects customer trust, and helps the team decide whether to roll back, iterate, or continue monitoring the redesigned AWS conversion experience.