Short answer
Experiment design creates tests with an explicit hypothesis, useful measures, and decision rules that produce actionable evidence.
About Experiment design
Experiment design is the ability to structure a test so its hypothesis, method, measures, and decision rule can reduce uncertainty about a choice.
Use this competency for
- Roles that test product assumptions before making or expanding investments.
- Work where a decision can be informed through controlled or structured evidence.
Do not use this competency for
- Decisions that cannot be tested ethically or within a useful decision window.
Important distinctions
Outcome measurement
Experiment design creates a test that can inform a decision. Outcome measurement evaluates whether an initiative produced its intended result.
Expectations by level
IC1
Individual contributor 1
With guidance, designs a bounded test for one explicit assumption. Handles a familiar method and defines what evidence would support the next decision.
Observable behaviors
- Writes a hypothesis that names the expected change and affected group.
- Selects a measure that relates directly to the tested assumption.
- Defines the decision rule before collecting results.
Examples
- Before testing a new prompt, records the expected completion change and the threshold for continuing.
- For a prototype session, uses the same task and evidence criteria with each participant.
IC2
Individual contributor 2
Independently designs tests for ambiguous product questions and chooses methods that fit the risk, audience, and decision window. Accounts for bias and alternative explanations.
Observable behaviors
- Chooses a test method based on the uncertainty rather than tool availability.
- Defines guardrail measures and checks for confounding changes.
- Documents limitations and avoids claiming more than the evidence supports.
Examples
- When testing onboarding guidance, separates the target action from unrelated traffic changes.
- After a mixed result, identifies which segment evidence is usable and which question remains open.
IC3
Individual contributor 3
Sets experiment practices across a team and guides tests with high uncertainty or broad consequences. Enables comparable evidence and prevents invalid conclusions from driving investment.
Observable behaviors
- Defines review standards for hypotheses, measures, and decision rules.
- Coordinates tests that could interfere across connected product areas.
- Challenges conclusions when design limits do not support the claimed decision.
Examples
- When teams test related changes, sequences exposure so one test does not contaminate another.
- Before a broad rollout test, requires a guardrail and stop rule for a material customer risk.