Design an experimentation dashboard for Support

Problem Statement Description

Product context: DoorDash is a local commerce and delivery platform; its products include restaurant delivery, DashPass, grocery and retail delivery, merchant tools, and dasher tools.

DoorDash Support runs experiments that can affect enterprise merchants, store operators, support agents, Dashers, and customers across a live local-commerce marketplace. In this interview, you are asked to define what an experimentation dashboard for Support should measure and how it should help teams interpret whether a change is improving the support experience without harming marketplace reliability, merchant outcomes, or unit economics.

Focus on the metrics and decision framework for experiments such as new support workflows, routing logic, automation, escalation paths, merchant self-serve tools, or agent-assist features. The dashboard should make it clear who is included in an experiment, what the primary success metric is, how outcomes are measured, and whether observed changes are trustworthy enough to inform a rollout decision.

You do not need to design the visual UI in detail. Instead, define the dashboard’s metric structure, instrumentation needs, segmentation, guardrails, and how different stakeholders—Support ops, product, data science, enterprise merchant teams, and marketplace operations—would use it to make decisions.

The experience should consider:

- Clear experiment context, including hypothesis, treatment/control definitions, exposure unit, start/end dates, sample size, and experiment status.

- Primary success metrics for Support, with precise numerators and denominators, such as resolution effectiveness, contact rate, time to resolution, escalation rate, reopen rate, or customer/merchant satisfaction.

- Enterprise merchant-specific cohorts, including merchant size, vertical, geography, store count, issue type, support channel, and integration complexity.

- Instrumentation requirements across support tickets, chat/phone/email interactions, merchant accounts, order events, refunds/credits, Dasher/customer impact, and agent actions.

- Guardrail metrics that detect negative marketplace effects, such as delayed orders, canceled orders, excessive refunds, merchant churn signals, Dasher friction, customer complaints, or support cost increases.

- Statistical and operational readiness indicators, including confidence, sample sufficiency, experiment health, logging gaps, contamination, and unusual traffic patterns.

- Decision usefulness for rollout, iteration, or rollback, including how results should be compared across cohorts and how trade-offs should be surfaced.

The goal is to describe a metrics dashboard that helps DoorDash evaluate Support experiments rigorously and operationally, especially for enterprise merchants where reliability, speed, accuracy, and business impact are critical.