Sources of bias in surveys
What bias means here
- Bias is a systematic tendency for an estimate to be too high or too low, built into the way the study was done.
- Two tests to apply to anything you want to call bias:
- Is it systematic? A one-off fluke is variability, not bias.
- Which way does it push? An answer that names a bias without stating the direction is incomplete.
Selection bias
- Selection bias arises when the way people are chosen makes some groups more likely to appear than others.
- Self-selection — participants volunteer. Over-represents strong opinion.
- Convenience — whoever was easy to reach. Over-represents whoever is at that place at that time.
- Undercoverage — part of the population is not on the frame at all.
- Survivorship — only those still present are surveyed. Asking current employees about workplace culture cannot reach the people who left because of it.
Non-response bias
- Non-response bias occurs when the people who do not reply differ from those who do.
- The key figure is the response rate:
- A high response rate is a strong defence; a low one turns a random sample into an effectively self-selected one.
- Ask who is likely to be missing and whether they differ on the variable:
- A survey about time pressure will lose the busiest people.
- A survey about illness will lose the sickest.
- A survey about a service will lose those who no longer use it.
- Non-response is not the same as undercoverage. In undercoverage they could never have been selected; in non-response they were selected and did not answer.
Question wording
- Leading questions point to an answer.
- "Do you agree that the council should stop wasting money on the cycleway?" is not measuring opinion on cycleways.
- Loaded words carry emotional weight — slash, reform, protect, waste, crisis.
- Double-barrelled questions ask two things at once.
- "Should the government reduce taxes and increase health funding?" — someone who wants one but not the other cannot answer.
- Assumed premises trap the respondent.
- "How much do you spend on takeaways each week?" assumes they buy takeaways.
- Unbalanced response options — offering excellent, very good, good, fair has three positive options and no negative one.
- Ambiguity — "Do you exercise regularly?" means different things to different people. "On how many of the last 7 days did you do at least 30 minutes of exercise?" does not.
Response and measurement bias
- Social desirability bias — people give the answer that reflects well on them. Consistently affects reported voting, exercise, charitable giving, alcohol consumption and recycling.
- Recall bias — people misremember, and misremember in patterned ways. Recent and dramatic events are over-reported.
- Interviewer effects — respondents adjust answers to the person asking. A supermarket employee in uniform asking about that supermarket's service gets kinder answers than an anonymous form.
- Order effects — earlier questions frame later ones. Asking about crime rates before asking about overall satisfaction with a neighbourhood lowers the satisfaction scores.
- Mode effects — the same question gets different answers by phone, online, and face to face. This matters when a report compares two surveys done by different methods.
Which biases a bigger sample fixes
- None of them. Every bias on this page is systematic. This is worth repeating because reports invariably respond to criticism by citing their sample size.
Worked ExampleFinding the bias, naming it, and stating the direction
A gym chain reports:
"We emailed our current members a satisfaction survey. 1 900 of 9 500 members replied. One question asked: 'How would you rate the excellent facilities at your gym?' with options Outstanding / Excellent / Very good / Good. 94% rated the facilities Very good or better, so our facilities meet members' needs."
Identify three distinct sources of bias, state the direction of each, and give an overall judgement of the conclusion.
Bias 1 — Survivorship / selection bias in the frame
Identify. Only current members were emailed.
Mechanism. Anyone who cancelled their membership because the facilities were inadequate — broken equipment, overcrowding, poor changing rooms — is not on the list. They have zero chance of selection, so this is undercoverage of exactly the group whose opinion would contradict the conclusion.
Bias 2 — Non-response bias
Identify. The response rate is
Mechanism. Four out of five members did not reply. Members who feel positively about their gym and visit often are more likely to open a gym email and engage with it; disengaged or lapsed-but-still-paying members are more likely to ignore it.
Bias 3 — Question wording
Identify. The question reads "How would you rate the excellent facilities…", and the four options are Outstanding / Excellent / Very good / Good.
Mechanism. Two separate faults compound:
- The word "excellent" inside the question is a leading term — it tells the respondent what the expected answer is before they choose.
- The scale is unbalanced: all four options are positive. A member who thinks the facilities are poor has no option to select, and will either pick the least positive option or abandon the survey.
A neutral replacement: "How would you rate the facilities at your gym?" with Very good / Good / Adequate / Poor / Very poor.
Overall judgement
Step 1 — Note that all three push the same way. Every fault inflates apparent satisfaction, so they compound rather than cancel. There is no plausible reading in which the true figure is higher than 94%.
Step 2 — Assess the conclusion.