Re-randomisation and conclusions
Comparing the two groups
- After the experiment, compare the distributions of the response for the two groups — usually with box plots.
- Look at the centres (medians) and the spread, and find the difference between the group centres — the observed difference.
- The key question: is that difference real, or just the luck of which units happened to land in which group?
The re-randomisation test
- Assume the treatment had no effect. Then each unit's outcome would have been the same whichever group it was placed in.
- Re-shuffle the group labels at random, over and over, and each time record the difference between the two groups.
- These build a re-randomisation distribution — the differences you would expect from the chance of allocation alone.
Making the call with the tail proportion
- See where the observed difference sits in that distribution.
- If it falls in the tail — few re-randomisations are as extreme — the difference is unlikely to be chance, which is evidence the treatment had an effect.
- If it sits in the middle — a difference that size is common under shuffling — the result is consistent with chance, so there is not enough evidence of an effect.
- A small tail proportion points to a genuine treatment effect; a large one does not.
Writing the conclusion
- State whether there is or is not evidence that the treatment caused a change — in context.
- Because the units were randomly allocated, a clear effect can be described as causal for the units in the experiment.
- Scope of inference: you may generalise the causal claim to a wider population only if the units were also a random sample from it. Otherwise the conclusion applies to these participants only.
In a reaction-time trial, participants were randomly allocated to a caffeine group or a water group. The caffeine group's median reaction time was ms faster than the water group's. The labels were then re-randomised times; only of those re-randomisations produced a difference of ms or more in favour of caffeine. State a conclusion.
Step 1 — Locate the observed difference
The observed difference of ms sits in the tail of the re-randomisation distribution — only of chance shuffles are that extreme.
Step 2 — Interpret the tail proportion
A tail proportion of is small, so a difference this large is unlikely to have arisen from the chance of allocation alone.
Step 3 — Conclusion with scope
There is strong evidence that caffeine caused a faster reaction time for the participants in this experiment (causal, because allocation was random). We should not generalise beyond these participants unless they were a random sample from a wider population.
Test yourself
Practice by grade
One question each at Achieved, Merit and Excellence. Have a go, then compare with the model answer.
After re-randomising the group labels many times, a student finds that a difference as big as the observed one occurs about of the time. What does this suggest about the treatment?
Explain why re-randomising the group labels lets you judge whether an observed difference is due to the treatment.
In a randomised experiment on 60 volunteers, a new revision method raised test scores by a median of 6 marks over the standard method. Re-randomisation gave a difference of 6 or more marks in only 3% of shuffles. Write a full conclusion, including the scope of what can be claimed.