Design an experimentation dashboard for Autopilot

Problem Statement Description

Product context: Tesla is an electric vehicle, energy, and software company; its products include EVs, charging, vehicle software, Autopilot/FSD features, energy storage, and solar products.

Tesla is running experiments on Autopilot features used by ride-hail operators who manage fleets of connected vehicles across many cities, routes, driver behaviors, and charging patterns. These experiments may involve changes to perception, driver-assistance behavior, alerts, handoff flows, routing interactions, or fleet-level controls, and the outcomes must be evaluated with a high bar for safety, reliability, and business impact.

Design an experimentation dashboard that helps Tesla product, engineering, safety, data science, and fleet operations teams understand whether an Autopilot experiment is performing as intended. The dashboard should support decision-making across pre-launch monitoring, active experiment reads, and post-experiment analysis without oversimplifying safety-critical outcomes.

Your focus is not to propose a specific Autopilot feature, but to define what the dashboard should measure, how metrics should be structured, how experiment results should be segmented, and how the product team should interpret results for ride-hail fleet use cases.

The experience should consider:

- Clear definitions for primary, secondary, and guardrail metrics, including denominators such as miles driven, trips, vehicles, active Autopilot sessions, disengagement opportunities, or driver interventions.

- Instrumentation needed from vehicles, Autopilot software, driver interactions, trip context, charging state, fleet operations systems, and incident reporting.

- Cohorts and cuts such as city, road type, weather, time of day, vehicle model, software version, driver tenure, fleet operator, trip density, and Autopilot usage level.

- Experiment design considerations, including assignment unit, exposure definition, sample size, statistical confidence, ramp stages, and handling of overlapping experiments.

- Safety and reliability guardrails, including collision risk signals, near-miss proxies, disengagements, harsh braking, driver alerts, latency, sensor issues, and manual takeover patterns.

- Business and operator-facing metrics such as vehicle utilization, trip completion, driver workload, customer ride experience, downtime, maintenance impact, and operational cost.

- Data quality checks for missing telemetry, delayed uploads, inconsistent event definitions, biased samples, and edge cases where vehicles are offline or operating in unusual conditions.

- Decision usefulness for different stakeholders, including when to continue, pause, roll back, expand, or investigate an experiment further.

The goal is to describe a metrics dashboard that enables Tesla to evaluate Autopilot experiments responsibly in a ride-hail fleet context, balancing autonomy progress with measurable safety, operational reliability, and fleet business outcomes.