ab-test-design - Design trustworthy online A/B experiments
Designs and trust-audits live online controlled experiments before final statistical inference.
Tags
Updated: 2026-10-03Capabilities
Typical Inputs
Typical Outputs
What this skill does
- Define decision metrics
- Choose randomization units
- Size experiments with MDE
- Apply variance reduction
- Pre-commit stopping rules
- Validate SRM and A/A tests
- Audit novelty and primacy
- Audit suspicious results
- Hand off trusted data
Inputs
- Experiment variant definitions
- Historical baseline metrics
- Historical variance estimates
- Randomization unit
- Guardrail metrics
- Pre-experiment covariates
- Stopping rule
- Live traffic or cases
Outputs
- Experiment design
- Sample size and duration estimate
- Variance reduction plan
- Validity audit
- Stopping plan
- Launch checklist
- Trusted analysis handoff
Requirements
- Live traffic or cases
- Random assignment capability
- Assignment and outcome logging
- Historical metric data access
- Pre-launch metric specification