Define success metrics for fraud alert experience serving delivery partners

Problem Statement Description

You are evaluating the fraud alert experience for delivery partners in a marketplace delivery product. Delivery partners may receive alerts when a trip, account activity, payment event, device change, identity signal, or behavioral pattern is suspected to be fraudulent or at risk. The business priority is improving recovery rate, meaning the product should help valid partners resolve issues, correct false positives, complete required actions, and return to normal earning activity when appropriate.

This is a metrics question. Your task is to define how success should be measured for this alert experience, not to redesign the full fraud system. The answer should clarify what “recovery” means, who is included in the measurement, when the recovery window starts and ends, and how the team would know whether alerts are effective, fair, actionable, and operationally sustainable.

The experience should consider:

- The denominator for recovery rate, including which fraud alerts, partner states, geographies, and severity levels are included or excluded.

- Instrumentation across alert delivery, partner viewing, comprehension, action completion, review outcomes, and return-to-work status.

- Cohorts such as new vs. tenured delivery partners, high-risk vs. low-risk alerts, language, market, device type, and prior fraud history.

- Funnel metrics that show where partners drop off after receiving an alert.

- Guardrails for fraud loss, false positives, partner trust, support burden, appeal quality, and repeat abuse.

- Time-based measures, including speed to recover, time to resolution, and freshness of fraud decisions.

- Decision usefulness: how the metric set would help product, risk, operations, and support teams decide whether to ship, iterate, or roll back changes.

The goal is to propose a clear, practical metrics framework that lets a product team evaluate whether the fraud alert experience increases legitimate recovery while protecting the marketplace from abuse and minimizing unnecessary disruption for delivery partners.