Design an experimentation dashboard for Autopilot at global scale

Problem Statement Description

Product context: Tesla is an electric vehicle, energy, and software company; its products include EVs, charging, vehicle software, Autopilot/FSD features, energy storage, and solar products.

Tesla is running Autopilot experiments across a global fleet, including vehicles used by ride-hail operators where utilization is high, routes are repetitive, and safety expectations are especially strict. You are asked to design an experimentation dashboard that helps product, engineering, safety, operations, and executive stakeholders understand whether Autopilot changes are improving the driving experience without creating unacceptable risk.

The dashboard must support decisions across software releases, model updates, regional rollouts, vehicle hardware differences, road environments, and operator cohorts. It should help teams compare experiment variants, detect regressions quickly, and separate true product impact from noise caused by geography, weather, traffic density, driver behavior, fleet mix, or data-quality issues.

Because this is a metrics-focused question, the emphasis is not on UI polish alone. The core challenge is defining the right metrics, denominators, instrumentation, cohorts, guardrails, and decision framework so Tesla can evaluate Autopilot experiments consistently at global scale.

The experience should consider:

- Primary success metrics for Autopilot experiments, including how each metric is defined and normalized.

- Safety and reliability guardrails that must be monitored before expanding an experiment.

- Clear denominators, such as miles driven, trips, interventions, disengagement opportunities, active Autopilot time, or route segments.

- Instrumentation needed from vehicles, sensors, software logs, driver actions, ride-hail operations, and post-trip events.

- Cohort breakdowns by region, road type, weather, vehicle model, hardware version, software version, operator type, and usage intensity.

- Experiment-readout needs, including confidence, sample size, duration, variant comparison, and anomaly detection.

- Data-quality checks for missing logs, delayed uploads, inconsistent labeling, sensor errors, and biased fleet coverage.

- Decision usefulness for rollout, rollback, further investigation, or escalation to safety and engineering teams.

Your goal is to frame a dashboard that enables trustworthy experimentation decisions for Autopilot at global scale, balancing product improvement, safety, operational efficiency, and confidence in the underlying data.