Design an A/B test to improve perfect order rate for premium quick-commerce users

Problem Statement Description

Blinkit wants to improve the “perfect order rate” for premium quick-commerce users, a segment that expects high reliability, fast delivery, accurate item availability, and low-friction issue resolution. In this context, a perfect order may involve multiple operational and customer-facing dimensions: the order is accepted, picked accurately, fulfilled without avoidable substitutions or missing items, delivered within the promised SLA, and completed without customer complaints or refunds.

You are asked to design an A/B test for a product or operational experience change aimed at improving this rate. The focus is not on inventing a full product roadmap, but on defining a rigorous experiment that can determine whether the proposed change meaningfully improves order quality for premium users without creating unacceptable trade-offs for cost, speed, picker workload, inventory accuracy, or customer trust.

Your answer should clarify what exactly is being tested, who is eligible for the test, how success will be measured, and how Blinkit should interpret the results. Consider the realities of instant commerce: local dark-store inventory, demand spikes, substitutions, picker and delivery-partner constraints, and the need for reliable instrumentation across ordering, picking, packing, dispatch, delivery, and post-order support events.

The experience should consider:

- A precise definition of “perfect order rate,” including numerator, denominator, exclusions, and whether it is measured per order, per user, or per order line.

- The premium-user cohort definition and whether eligibility should depend on geography, store maturity, order frequency, subscription status, or historical reliability expectations.

- The experiment unit of randomization, such as user, order, store, or geography, and the risks of spillover between treatment and control.

- Instrumentation needed across inventory availability, substitution flows, picker actions, delivery SLA events, cancellations, refunds, complaints, and customer feedback.

- Primary metric, secondary diagnostics, and guardrail metrics such as delivery time, fill rate, substitution acceptance, support contacts, picker workload, delivery-partner utilization, refunds, and contribution margin.

- Segmentation cuts that would make the results actionable, such as category type, time of day, demand surge periods, store density, basket size, and new versus repeat premium users.

- Experiment duration, sample-size considerations, novelty effects, seasonality, and operational readiness before ramping exposure.

- Decision rules for shipping, iterating, or rolling back based on statistical confidence, business impact, customer experience, and operational cost.

The goal is to demonstrate how you would create a trustworthy metrics and experimentation framework for a high-expectation quick-commerce segment, ensuring that any improvement in perfect order rate is real, measurable, scalable, and useful for product and operations decision-making.