Conversion in Search fell after a redesign. How would you investigate

Problem Statement Description

Product context: Google is a consumer technology, ads, AI, and cloud company; its products include Search, YouTube, Android, Maps, Gmail, Chrome, Google Play, Workspace, and Google Cloud.

Google Search has recently launched a redesign, and shortly after rollout, a key conversion metric has declined. In this case, “conversion” may refer to the intended downstream action for Search users, such as clicking a result, engaging with a specialized result module, completing a developer-focused query journey, or reaching a useful destination from the search results page.

You are asked to investigate the drop as a product manager responsible for Search quality and user experience. The context includes a global product surface, high-volume query traffic, multiple platforms, and a developer user segment that may rely on Search for technical documentation, APIs, code examples, troubleshooting, and enterprise workflows.

Your task is not to propose a redesign immediately, but to structure a root-cause investigation. You should clarify the anomaly, validate whether the metric movement is real, identify which users or experiences are affected, generate hypotheses, and explain how you would use data and product judgment to decide what to do next.

The experience should consider:

- How conversion is defined, including numerator, denominator, eligible sessions, query types, and whether the metric changed after the redesign.

- Whether the decline is statistically significant and isolated to the redesign period, rollout group, platform, geography, browser, query category, or developer cohort.

- Instrumentation checks, including event logging changes, tracking breakages, attribution windows, bot filtering, consent effects, and experiment assignment integrity.

- Segmentation by surface and journey, such as desktop vs. mobile, logged-in vs. logged-out, organic results vs. rich modules, AI-generated answers, ads, or developer-specific result types.

- User-behavior signals that could explain friction, such as lower result clicks, higher reformulation, faster abandonment, increased pogo-sticking, slower page load, or reduced trust in result quality.

- Hypotheses related to the redesign, including layout changes, ranking visibility, module placement, latency, accessibility, information density, or altered prominence of trusted developer sources.

- Evidence needed to distinguish product issues from external factors such as seasonality, traffic mix shifts, competitive behavior, ecosystem changes, or changes in developer demand.

- Mitigation and prevention considerations, including rollback options, targeted fixes, experiment holdouts, monitoring dashboards, and post-launch guardrails.

The goal is to demonstrate a clear RCA approach for a Google-scale Search product: frame the metric drop precisely, separate measurement issues from real user impact, narrow the affected area through segmentation, prioritize likely causes with evidence, and recommend a disciplined path toward mitigation without jumping prematurely to a solution.