Comparing two percentages and making the call

Why you cannot just compare the two intervals

• A report gives two percentages and claims one is bigger. To judge that claim you need the margin of error for the difference, not for either percentage on its own.
• The difference of two estimates carries both uncertainties, so its margin of error is larger than either individual margin of error.
• Which rule you use depends on where the two percentages came from, and getting this right is one of the most reliable sources of marks in the standard.

Case 1 — two percentages from the same group

• This is comparing two response categories within one survey: Party A against Party B in the same poll, or "agree" against "disagree" among the same respondents.
• The two percentages are linked: they are shares of the same total, so if one rises the other must fall. That connection makes the difference more variable, not less.
• Rule:

$$\text{MoE(difference)} = 2 \times \text{MoE}$$

• Where $\text{MoE} \approx 1/\sqrt{n}$ for the single poll.

Case 2 — the same percentage in two independent groups

• This is comparing one category across two separate samples: this month's poll against last month's, men against women (as two separate subgroups), Auckland against Canterbury.
• The two estimates come from different people, so the estimates are independent.
• Rule:

$$\text{MoE(difference)} = 1.5 \times \frac{\text{MoE}_1 + \text{MoE}_2}{2}$$

• If the two groups are the same size, this simplifies to $1.5 \times \text{MoE}$.
• Note that subgroups have their own sample sizes. If a poll of 1 000 reports separately on 400 men and 600 women, use $n = 400$ and $n = 600$ — not 1 000.

Making the call

• Once you have the margin of error for the difference, the decision rule is one line:

If the observed difference is bigger than the margin of error for the difference, you can call it. If not, you cannot.

• Write the conclusion in three parts, in context:

  • the observed difference
  • the margin of error for the difference
  • the call, naming the population.

• Both possible conclusions have correct wording:

  • Difference exceeds the margin of error: "It is likely that a higher proportion of Auckland residents than Canterbury residents support the proposal."
  • Difference does not exceed it: "We cannot tell which group has the higher proportion — the difference could be due to sampling variability alone."

• Never say "there is no difference." Failing to detect a difference is not evidence that none exists; it means the survey was not precise enough to settle the question.

Subgroups are less precise than the headline

• Reports constantly headline a subgroup finding — "young people are far more likely to…" — using the whole survey's margin of error.
• If the subgroup has $n = 150$, its margin of error is $1/\sqrt{150} \approx 8.2\%$, not the 3.2% quoted for the full sample of 1 000.
• Spotting this is one of the highest-value criticisms available: the smaller the group a report is excited about, the less reliable its number.