Diagnose a sudden drop in workflow completion for marketplace quality score

Problem Statement Description

A marketplace provides local merchants with a quality score workflow that helps them understand and improve how their listings, service standards, fulfillment, responsiveness, reviews, or policy compliance are evaluated. Merchants may use this workflow to review score drivers, acknowledge issues, submit updates, complete recommended actions, or unlock better visibility and trust signals in the marketplace.

Recently, workflow completion for local merchants dropped suddenly. The task is to diagnose what changed before proposing any fixes. Treat this as a root-cause analysis problem focused on separating a real merchant behavior change from measurement issues, product regressions, traffic mix shifts, policy changes, or operational dependencies.

You should assume the workflow has multiple steps and may vary by merchant segment, category, geography, device, acquisition channel, score status, and eligibility state. The investigation should clarify where the drop occurred, who was affected, when it started, and whether the issue is isolated to the quality score experience or part of a broader marketplace trend.

The experience should consider:

- The exact definition of “workflow completion,” including start event, required steps, end event, denominator, and whether repeat attempts are counted.

- Timeline of the anomaly, including release dates, experiment launches, policy updates, backend changes, notification campaigns, or merchant support changes.

- Funnel segmentation by step, device, browser/app version, geography, merchant category, merchant size, quality score band, new vs. existing merchants, and traffic source.

- Instrumentation checks to confirm whether tracking, event firing, eligibility logic, or data pipelines changed around the same time.

- Product and UX hypotheses such as broken entry points, confusing score explanations, new friction, blocked submissions, page load issues, localization problems, or inaccessible design.

- Marketplace and merchant-context hypotheses such as seasonality, demand shocks, category-specific policy changes, reduced perceived value, or merchant distrust of the score.

- Evidence needed to distinguish correlation from causation, including logs, funnel data, session replays, support tickets, merchant feedback, experiment readouts, and operational dashboards.

- Immediate mitigation and prevention considerations, including severity assessment, merchant impact, rollback criteria, monitoring gaps, and communication needs.

The goal is to present a structured RCA approach that quickly narrows the problem, validates whether the drop is real, identifies the most likely root cause, and defines what evidence would be needed before recommending corrective action.