Topics

The randomisation test and the conclusion

The question a randomisation test answers

Your experiment produced a difference between the two groups. Before claiming the treatment did it, you must rule out the obvious alternative:

Could a difference this large have arisen purely from the luck of the random allocation, even if the treatment had no effect at all?

A randomisation test answers exactly that question, and it does so by simulating the experiment as if the treatment did nothing.

How it works

The logic: if the treatment had no effect, then each unit's result would have been the same whichever group it had been allocated to. The labels "treatment" and "control" would be arbitrary.
The procedure, carried out by the software:
- Pool all the results, ignoring which group they came from.
- Randomly re-allocate them into two groups of the same sizes as the original.
- Calculate the difference between the two group statistics.
- Repeat — typically 1 000 times or more.
- The 1 000 differences form the re-randomisation distribution: what chance alone produces when the treatment does nothing.

The tail proportion

Count how many of the re-randomised differences are at least as extreme as the one you actually observed, and divide by the number of repetitions:

$\text{tail proportion} = \frac{\text{number of re-randomisations at least as extreme as the observed difference}}{\text{total number of re-randomisations}}$

What it means: the proportion of the time chance alone would produce a difference as large as yours, if the treatment did nothing.

Interpreting it

Tail proportion	Interpretation
Large (say above 0.10)	Chance alone readily produces a difference this size — the experiment provides no evidence that the treatment had an effect
Small (say below 0.05)	Chance alone rarely produces a difference this size — there is evidence the treatment had an effect
In between	Weak or suggestive evidence; say so rather than forcing a verdict

These are guidelines, not a rule. The standard asks you to assess the strength of the evidence, not to apply a cut-off. Describe the strength in words and explain what the number means.

Wording the inference

When the tail proportion is small:
- "Only 12 of the 1 000 re-randomisations produced a difference as large as the 3.4 words observed, giving a tail proportion of 0.012. This means chance alone would produce a difference this large only about 1.2% of the time. There is therefore strong evidence that listening to music with lyrics caused a reduction in the number of words recalled, for the students in this experiment."
When it is large:
- "312 of the 1 000 re-randomisations produced a difference at least as large as the 0.6 words observed, a tail proportion of 0.312. Chance alone would produce a difference this size about 31% of the time, so the experiment provides no evidence that the treatment had an effect."

Two things to avoid

Do not say the treatment "had no effect" when the tail proportion is large. Say the experiment provides no evidence of an effect. A real but small effect could easily go undetected with a small experiment.
Do not say the result is "proved." Say the evidence is strong or weak.

What you can and cannot conclude

Because you randomly allocated, you can make a causal claim: the treatment caused the difference for the units in this experiment.
Because you did not randomly select your units from a wider population, you cannot generalise beyond them. A class experiment supports a conclusion about that class, not about all New Zealand students.
Saying both of these explicitly is one of the clearest signals of understanding available in this standard.

The conclusion

A conclusion that scores well:
- Answers the investigative question
- Reports the observed difference with summary statistics from both groups
- States the tail proportion and interprets it as strength of evidence
- Makes the causal claim, correctly limited to the units studied
- Reports any issues that arose during the experiment and their likely effect
- Discusses sources of variation and how the design handled them
- Considers other relevant variables and suggests improvements

Tips and Tricks

Report the tail proportion as a proportion and explain it in words. "0.012, meaning chance alone would produce a difference this large only about 1.2% of the time" — the explanation is what earns the mark.

Never write "the treatment had no effect." Write "the experiment provides no evidence of an effect", and note that a small real effect could go undetected.

State the causal claim and its limits in the same breath: "the treatment caused this difference for the students in this experiment, but because they were not randomly selected, this cannot be generalised to other students."

Report the issues that arose. A participant who misunderstood, a failed timer, an interruption — these are a listed component of the standard, and reporting them honestly with their likely effect gains credit.

Return to your sources-of-variation table in the conclusion and say whether the design handled each one adequately in practice. That closes the loop on the Excellence criterion.

Worked ExampleAnalysing and concluding

Continuing the music-and-memory experiment. Results:

	Music (n = 16)	Silence (n = 16)
Median words recalled	9.5	12.0
Mean	9.4	12.1
IQR	3.5	3.0

The observed difference in means is 2.7 words (silence − music). A randomisation test with 1 000 re-randomisations produced 18 differences at least as large as 2.7.

One student's headphones failed part-way through and their data were recorded with a note.

Write the analysis and conclusion.

Step 1 — Describe the results

Step 2 — Carry out and report the randomisation test

Step 3 — Interpret the tail proportion

Step 4 — State the causal claim with its limits

Why a causal claim is justified. Students were randomly allocated to the two conditions, so the two groups were similar on average in memory ability, tiredness, motivation and every other characteristic — including those that could not be measured. The only systematic difference between the groups was the listening condition, so the difference in recall can be attributed to it.

Why it cannot be generalised. The 32 students were not randomly selected from any wider population — they were the students available in this class. The conclusion therefore applies to these students, and cannot be extended to all Year 13 students, or to students generally, without a study using random selection.

This distinction is important and often confused: random allocation earns the causal claim; random selection would have earned the generalisation, and it was not present.

Step 5 — Report the issues that arose

One student's headphones failed part-way through the task. This student was in the music condition and effectively completed part of the task in near-silence, so their result is not a clean measurement of the music condition.

How this was handled: the issue was recorded at the time and the student's data were retained in the analysis, with the problem noted. Retaining it is the conservative choice here: the student's partial exposure to music would tend to make their recall closer to the silence group's, which would reduce the observed difference rather than inflate it. In other words, the issue works against the conclusion drawn, so the evidence is if anything slightly stronger than the tail proportion suggests.

Removing the data point would have been the wrong response without stating it, since selectively excluding results that do not fit is exactly how experiments are unconsciously biased.

Step 6 — Close the loop on the design

Return to the sources of variation you identified when planning, and say how each one held up in practice. The single most valuable sentence is the one naming what the design could not control:

Then propose the improvement that would test the mechanism, rather than a generic "use more people":

Step 7 — The conclusion

The conclusion pulls the pieces together in a few sentences. It does not restate the whole analysis:

Why this reaches Excellence