Sampling, representativeness and bias
From sample to claim
- Almost every report measures a sample and then makes a claim about a whole population.
- That claim is only trustworthy if the sample is representative — if it looks like the population in the ways that matter.
How was the sample chosen?
- Random sample — everyone in the population has an equal chance of being picked. This is the gold standard: it avoids systematic bias and lets the sample stand in for the population.
- Convenience sample — whoever is easy to reach (the first 30 people outside a shop). Usually not representative.
- Self-selected (voluntary response) — people choose to take part (a website poll, a text-in). Strongly biased: only those with strong opinions or time respond.
Sources of bias (non-sampling errors)
- Self-selection / selection bias — the way people enter the sample skews who is in it.
- Non-response bias — those who do not reply may differ from those who do, so the responses are lop-sided.
- Question wording — a leading or loaded question pushes people toward an answer ("Don't you agree that…?").
- Measurement bias — the way the response is collected distorts it.
Sample size
- A larger sample gives a more reliable result — but a big biased sample is still biased. Size does not fix a bad sampling method.
The questions to ask of any report
- Who was in the sample, and how were they chosen?
- How many were sampled?
- Who was left out, and could that skew the result?
A news website runs an online poll: "Should the council build more cycleways?" Of 4,000 people who clicked, 78% said yes. The article concludes "most residents support more cycleways." Evaluate this.
Step 1 — Identify the sampling method
The sample is self-selected (voluntary response) — only people who visited the site and chose to click took part.
Step 2 — Explain the bias
People who click on such a poll tend to hold strong views on the topic, and website visitors are not a cross-section of all residents. So the 78% reflects the opinions of a keen minority, not the population — this is selection bias.
Step 3 — Effect on the conclusion
The conclusion "most residents support it" is not justified: the sample cannot be generalised to all residents, no matter that 4,000 responded.
Step 4 — Improvement
Test yourself
Practice by grade
One question each at Achieved, Merit and Excellence. Have a go, then compare with the model answer.
A reporter interviews the first 25 people leaving a gym and concludes New Zealanders exercise regularly. Name the sampling method and one reason it is biased.
A magazine mails a survey to subscribers and reports results from the 12% who replied. Explain how non-response could bias the results.
A company surveys its own customers and reports '92% are satisfied,' concluding its product is the best on the market. Evaluate this fully and suggest a better method.