Short answer
Experiment analysis estimates what a controlled change caused by checking assignment, exposure, outcomes, uncertainty, and practical importance together.
About Experiment analysis
Experiment analysis analyzes controlled tests to estimate effects, uncertainty, and practical significance. It protects causal interpretation by checking design execution as well as the final estimate.
Use this competency for
- Roles that design, validate, or analyze randomized product, marketing, or operational tests.
- Work where a controlled comparison informs whether to launch, stop, or revise a change.
Do not use this competency for
- Use statistical reasoning when the work is not centered on controlled experiments and their causal interpretation.
Important distinctions
Statistical reasoning
Statistical reasoning covers methods and uncertainty broadly, while experiment analysis focuses on controlled tests and causal effects.
Data analysis
Data analysis may describe differences in observed data, while experiment analysis checks whether a controlled change caused an outcome.
Expectations by level
IC1
IC1: Defined experiments
Analyzes a well-defined experiment with guidance using the approved assignment, exposure, and outcome definitions. Checks execution and reports effect and uncertainty.
Observable behaviors
- Verifies sample counts, assignment balance, and experiment dates before reading outcomes.
- Calculates the approved primary metric and uncertainty from the assigned groups.
- Separates the planned result from exploratory cuts in the report.
Examples
- Analyzes a checkout test, finds an assignment imbalance, and pauses interpretation until instrumentation is checked.
- Reports a small conversion estimate with a wide interval and avoids declaring a winner.
IC2
IC2: Experiment decisions
Independently plans and analyzes team-level experiments with ambiguous tradeoffs. Defines decision criteria, diagnoses validity threats, and explains practical consequences.
Observable behaviors
- Defines primary metrics, guardrails, unit of assignment, and analysis plan before results are known.
- Checks exposure, attrition, interference, and repeated peeking for threats to interpretation.
- Translates estimates and uncertainty into launch, iterate, or collect-more-data options.
Examples
- Designs a notification test with user-level assignment to prevent messages crossing groups.
- Finds that a pricing test changes refunds as well as purchases and presents both effects in the decision.
IC3
IC3: Experimentation standards
Sets experimentation methods used across teams and leads high-impact tests with complex assignment or long-term effects. Reviews whether causal claims match the design.
Observable behaviors
- Defines shared standards for metric choice, power planning, validity checks, and result reporting.
- Designs experiments for clustered assignment, network effects, or delayed outcomes.
- Reviews portfolios of tests for conflicting treatments and repeated decision errors.
Examples
- Creates an experimentation review that catches overlapping treatments before two teams launch.
- Designs a region-level operations test and accounts for cluster variation in planning and analysis.