Diagnose a sudden drop in decision quality for AI sales assistant

Problem Statement Description

You are investigating a sudden decline in decision quality for mobile-first users of an AI sales assistant. The product helps sales representatives, account managers, or field teams make customer-facing decisions such as prioritizing leads, preparing outreach, choosing next-best actions, summarizing account context, and responding to buyer signals. The issue is isolated or most visible among users who primarily interact through mobile devices.

Your task is to diagnose what may have changed before proposing fixes. Treat “decision quality” as an outcome that may be measured through user actions, sales workflow effectiveness, AI recommendation acceptance, downstream conversion, manager review, or user feedback. The RCA should distinguish whether the drop reflects a real degradation in AI assistance, a measurement or instrumentation issue, a mobile experience problem, a user-segment shift, or an external business change.

Assume the product operates at scale and depends on multiple systems: mobile app UX, recommendation models, customer/account data pipelines, CRM integrations, question or policy layers, ranking logic, notifications, and analytics instrumentation. You should frame the anomaly, identify the most important cuts of data, generate testable hypotheses, and explain how you would validate or rule them out.

The experience should consider:

- How “decision quality” is defined, measured, and whether the metric denominator or logging changed.

- Whether the decline is limited to mobile-first users, specific devices, app versions, regions, sales roles, account types, or workflow stages.

- Recent changes to the AI model, questions, ranking logic, CRM data sync, mobile app release, notification behavior, or permissions.

- Instrumentation checks for missing events, duplicate events, delayed ingestion, attribution changes, or changes in confidence-score capture.

- User behavior signals such as recommendation views, acceptance, edits, overrides, session completion, time-to-decision, and abandonment.

- Data-quality hypotheses involving stale account data, incomplete mobile context, missing customer history, or degraded third-party integrations.

- Evidence needed to separate AI accuracy issues from UX friction, trust erosion, sales-process changes, or seasonality.

- Short-term containment, stakeholder communication, monitoring, and prevention mechanisms once the root cause is confirmed.

The goal is to demonstrate a structured RCA approach: clearly frame the anomaly, segment the impact, verify the data, prioritize plausible hypotheses, identify the evidence required, and only then move toward mitigation and prevention.