Critiquing statistical claims
A checklist for evaluating a report
- When a report or headline makes a statistical claim, work through:
- Who was sampled, and how? Was it random and representative, or self-selected?
- How big was the sample? Small samples are unreliable; check the margin of error.
- What type of study was it? Only experiments justify causal claims.
- Who ran or funded it? A vested interest can bias the design or reporting.
- Does the graph mislead? A truncated axis or odd scale exaggerates differences.
Match the claim to the evidence
- The strength of the conclusion must match the quality of the data.
- A survey supports statements about an association or an estimate; only a randomised experiment supports "causes".
A supplement company reports: "In our study, 8 out of 10 people felt more energetic after taking our pills for a week." Evaluate how much this claim can be trusted.
Step 1 — Sample size and selection
Only 10 people were studied — far too few for a reliable result, and it is unclear how they were chosen (likely volunteers or customers, i.e. self-selected).
Step 2 — Study design
There is no control group taking a dummy pill, so the reported effect could be the placebo effect rather than the pills.
Step 3 — Vested interest
The company funded its own study, giving a clear incentive to report a positive result.
Step 4 — Conclusion
Taken together, the claim is not trustworthy. A large, randomly chosen sample, a placebo control group, and independent funding would be needed to support it.
Test yourself
Practice by grade
One question each at Achieved, Merit and Excellence. Have a go, then compare with the model answer.
A graph's vertical axis starts at 90 instead of 0. State one problem this causes.
A website claims '90% of dentists recommend our toothpaste', based on a survey of 20 dentists paid by the company. Give two reasons this claim may be unreliable.
A newspaper reports that a town's crime 'soared 50%' after a new policy, based on comparing one month before and one month after. Critically evaluate this claim.