How political polls are actually conducted
Why polls get their own page
- Polls are the single most common report context in this standard, and they come with conventions you are expected to recognise.
- A poll is a survey with three extra complications: the population is voters, the quantity is a set of percentages that must add to 100, and the result is published as a change since last time.
What a published methodology statement contains
- Reputable New Zealand polls publish a statement alongside the numbers. You should be able to read one and say what it does and does not protect against.
| What it says | What to check |
|---|---|
| "n = 1 000 eligible voters" | Gives a margin of error near |
| "Conducted 11–17 March" | Did something happen mid-field that would change answers? |
| "Mobile and landline, plus online panel" | Mixed modes — mode effects apply |
| "Weighted by age, gender, region, education" | Fixes proportions between groups, not differences within them |
| "Undecided voters excluded" | The published percentages are of decided voters only |
| "Maximum sampling error 3.1% at 95% confidence" | Applies to a percentage near 50%, and only to sampling variability |
The undecided problem
- Most published party percentages are calculated after removing undecided respondents.
- If 15% are undecided and a party polls 30% of the remainder, that party has the support of of all respondents.
- Why this matters for evaluation:
- The denominator has changed, so the percentages describe decided voters, not voters.
- If the undecided group grows, every party's percentage can rise without a single person changing their mind.
- Undecided voters are not a random subset — they are disproportionately less engaged, and historically break differently from decided voters.
Percentages that must sum to 100
- Party support figures are not independent of each other. If one party gains 3 points, some other party or parties must lose 3.
- This is why you cannot treat two party percentages from the same poll as two separate estimates when comparing them — a point developed fully in the next-but-one page.
Change since the last poll
- Headlines are almost always about movement: "Support for X up two points."
- Two questions dismantle most such headlines:
- Is the movement bigger than the margin of error for a change? A two-point move in a poll with a 3.1% margin of error is not evidence of any change at all.
- Were the two polls conducted the same way? A change of pollster, mode or question order can produce a shift with no change in opinion. Comparing across pollsters is comparing two different measuring instruments — often called a house effect.
What a poll cannot do
- It cannot predict an election. It measures stated intention now, among those who responded, on the assumption they will turn out to vote. Turnout is not modelled by a simple poll and varies sharply by age.
- It cannot detect its own bias. If the people who answer polls differ politically from those who do not, no amount of weighting or sample size will reveal it.
- It cannot measure strength of support. A soft "probably X" and a certain "definitely X" count identically.
Evaluating a poll report — the standard checklist
- Population: eligible voters, decided voters, or all adults?
- Frame and mode: who could be reached?
- Response rate: usually low, and usually unpublished.
- Undecideds: included in the denominator or removed?
- Margin of error: quoted, and is it the right one for the comparison being made?
- Comparison: same pollster, same method, same question?
Worked ExampleReading a poll report properly
A news outlet publishes:
"In our latest poll, support for the Green Party rose from 9% to 11%, while support for New Zealand First fell from 6% to 5%. The Greens are clearly gaining ground at New Zealand First's expense. Poll of 1 000 eligible voters, maximum margin of error 3.1%. 14% were undecided and are excluded from the figures above; last month 9% were undecided."
Evaluate the report's conclusion.
Step 1 — Test the movements against the margin of error
The Greens: 9% to 11%, a 2-point rise. The quoted maximum margin of error is 3.1%, which applies at 50%. At a percentage as low as 10% the margin of error is actually smaller — but even taking the conservative 3.1%, a 2-point movement is well inside it.
New Zealand First: 6% to 5%, a 1-point fall. Comfortably inside the margin of error.
Step 2 — Examine the denominator
The undecided proportion went from 9% to 14% — a rise of 5 percentage points, which is itself larger than either party movement being reported.
Because undecideds are excluded, the published percentages are shares of decided voters, and the decided group shrank from 91% of respondents to 86%.
Work out what this does. Suppose the Greens' support among all respondents were completely unchanged. As a share of all respondents:
So even in raw terms there is a small apparent rise — but the point runs the other way too: a party's published percentage can rise purely because the pool it is a share of got smaller. With 5% more voters moving to undecided, small upward drifts in the published figures are expected across the board.
Step 3 — Examine the causal claim
The report says the Greens are gaining ground "at New Zealand First's expense".
This asserts a transfer between two specific parties. A poll of this kind is a series of independent snapshots — different people are sampled each time, and no respondent is tracked from one poll to the next.