Conversion in AWS fell after a redesign. How would you investigate

Problem Statement Description

Product context: Amazon is a commerce, logistics, media, devices, and cloud company; its products include Marketplace, Prime, Prime Video, Alexa devices, ads, fulfillment, and AWS. AWS is Amazon's cloud platform; its products include compute, storage, databases, analytics, networking, security, machine learning, and developer tools.

AWS has recently launched a redesign of a key acquisition or onboarding experience used by small merchants evaluating cloud services for their commerce operations. After launch, the team observes a meaningful drop in conversion. Conversion may refer to a business-critical step such as account creation, starting a free tier trial, selecting a service, completing billing setup, or successfully launching the first workload.

You are asked to investigate the drop as a product leader in an RCA interview. The focus is not to propose a redesign immediately, but to structure how you would diagnose whether the decline is real, where it is happening, who is affected, what changed, and what evidence would determine the next action.

The context is AWS, where customers may be technical or semi-technical small-business owners, developers, agencies, or merchant operations teams. The experience may involve complex pricing, identity setup, billing, service selection, documentation, support, and trust concerns. The redesign may have changed UI flows, copy, information architecture, page performance, eligibility rules, tracking, or traffic allocation.

The experience should consider:

- How to define the conversion metric precisely, including numerator, denominator, funnel stage, time window, and whether the same definition applied before and after the redesign.

- How to validate that the anomaly is real by checking instrumentation, event logging, attribution, traffic mix, experiment assignment, bot/internal traffic, and data pipeline changes.

- How to segment the decline across merchant size, geography, device, browser, acquisition channel, new versus returning users, technical sophistication, service category, and pricing/free-tier eligibility.

- How to inspect the funnel step by step, from landing page and sign-up through identity verification, billing, service discovery, documentation, and first successful activation.

- What hypotheses could explain the drop, including usability friction, slower page load, confusing pricing or copy, missing trust cues, increased form burden, broken integrations, support deflection, or changed ranking of AWS services.

- What evidence would help distinguish product friction from external causes such as seasonality, marketing campaign changes, competitor activity, sales outreach changes, or broader cloud demand shifts.

- How to decide on mitigation actions such as rollback, traffic reduction, targeted fixes, customer support escalation, or additional experiments without prematurely assuming the root cause.

- How to prevent recurrence through better pre-launch validation, monitoring, alerting, experiment design, and ownership across product, engineering, analytics, design, marketing, and support.

Your goal is to present a rigorous investigation plan that narrows the problem from a broad post-redesign conversion decline to a prioritized set of evidence-backed root causes, while protecting small merchant customers and AWS business outcomes during the investigation.