Experimental design principles

What makes something an experiment

• In an experiment, the researcher applies a treatment and decides who gets which one.
• In an observational study, the researcher only observes what people or things already do.
• That single difference is why experiments can support causal claims. If you allocate treatments at random, the groups start out alike in every respect — measured and unmeasured — so a difference in outcomes can be attributed to the treatment.

The vocabulary

• Get these named explicitly in your report. The standard lists "determining treatment and response variables" and "selecting experimental units" as separate required components.

Random allocation — the heart of the design

• Random allocation is not the same as random selection.
  • Random selection decides who is in the study, and it determines whether you can generalise to a wider population.
  • Random allocation decides who gets which treatment, and it is what allows a causal conclusion.
• What random allocation achieves: it balances all characteristics between the groups on average — including ones you never thought of and could not measure. No other technique does this.
• How to do it properly:
  • Assign each unit a number.
  • Use a random number generator or draw numbers from a hat.
  • Allocate to groups in the resulting order.
• What does not count as random: letting people choose, allocating by which class they are in, alternating down a list, or "just mixing them up".

Sources of variation, and how to deal with them

• The Excellence criterion names this directly: discussing how possible sources of variation were dealt with during the design phase.
• There are three kinds, and each has its own remedy:

• Controlling means holding a variable constant for everyone: same room, same time of day, same equipment, same instructions.
• Blocking means grouping similar units together and randomising within each block — if you know age matters, split into age bands and randomise within each.
• Random allocation handles the rest. This is the argument to make in your report: "I could control for X and Y, but I could not measure Z, so I relied on random allocation to balance it between the groups."

Other design principles

• Replication. Each treatment must be applied to enough units for the result to be more than chance. One unit per treatment tells you nothing.
• Blinding. Where possible, participants should not know which treatment they received, since expectation affects outcomes. Double-blinding — where the person measuring also does not know — removes measurement bias too.
• Standardised measurement. The response must be measured the same way for every unit: same instrument, same procedure, same person if possible.
• A control group. Without a comparison, a change cannot be attributed to anything.

Considering other relevant variables

• Before running the experiment, list the things other than your treatment that could affect the response, and say what you did about each.
• For a school experiment on, say, reaction time:
  • Time of day — control it: test everyone in the same period.
  • Caffeine or sleep — cannot control, so randomise and note it as a limitation.
  • Practice effect — control it by giving everyone the same number of practice attempts first.
  • Hand used — control by requiring everyone to use their dominant hand.
• Writing this list, with a remedy against each item, is the single most valuable page of an Excellence report.