Putting it together: critiquing a whole report
What a full-report question looks like
- The longest question in the paper gives you a report of several paragraphs — often with a table or graph — and asks you to evaluate it, sometimes with a specified number of points.
- These questions carry the Excellence marks, and they are lost far more often to poor structure than to poor knowledge.
A structure that works every time
- 1. Read the conclusion first. Everything you write must bear on it. Underline it before you read the rest.
- 2. Note the population in the conclusion, and the population actually studied.
- 3. Identify the study type. Survey, poll, experiment, observational study. This settles what language the conclusion is entitled to.
- 4. Work down the enquiry cycle, noting features as you go: frame, selection, response rate, question wording, measurement, analysis, display.
- 5. Choose your two or three strongest points. Not your first three.
- 6. Develop each fully, in the same three-move pattern.
- 7. Finish with an overall judgement and, if the question invites it, what would fix the report.
The three-move pattern for every point
- This is the single most useful thing to memorise for this standard. Each point you make has three moves:
| Move | What it does | Grade it reaches |
|---|---|---|
| Identify | Name the feature of this report | Achieved |
| Justify | Quote the evidence or process from the report and say why it is a problem, with a direction | Merit |
| Evaluate | Say what it means for this conclusion, in this context | Excellence |
- Written out, a complete point looks like this:
"The survey was sent only to current members (identify). Anyone who cancelled because of the facilities is therefore excluded from the frame entirely, so the sample has already been filtered for satisfaction and the 94% will be an overestimate (justify). Because the report's conclusion is specifically that the facilities meet members' needs, the one group whose experience would contradict it is the group that cannot be reached — so the evidence cannot support the claim in either direction (evaluate)."
Choosing which points to make
- Prioritise faults that affect the conclusion. A minor wording quibble in a question the conclusion does not rest on is not worth writing about.
- Prefer faults you can develop. A criticism you can carry through all three moves beats two you can only name.
- Cover different stages. Three points all about sampling look thinner than one about sampling, one about measurement and one about the inference.
Writing about what the report does well
- A report is not required to be worthless. Noting a genuine strength — "the response rate of 78% is high, so non-response bias is unlikely to be substantial" — demonstrates judgement, and judgement is what Excellence rewards.
- The best answers often take this shape: the design is sound in these respects, so the estimate itself is credible; the failure is in the conclusion drawn from it.
Phrases that carry marks
- "The conclusion is about X, but the sample was drawn from Y."
- "This will cause the figure to be an over/underestimate, because…"
- "The margin of error accounts only for sampling variability, and makes no allowance for…"
- "The difference of D is smaller than the margin of error for the difference of M, so no call can be made."
- "Participants chose their own group, so the difference may be caused by [named confounder] rather than the treatment."
- "The report can legitimately claim…, but not…"
- "To support the stated conclusion, the study would need to…"
Phrases that earn nothing
- "The sample was biased." — Which bias? Which direction?
- "Correlation does not imply causation." — Which alternative explanation, and why here?
- "The sample was too small." — Compared with what? Calculate the margin of error instead.
- "They should survey more people." — Not if the problem is bias.
- "The report is wrong." — Reports are evaluated, not marked right or wrong.
Worked ExampleA full report critique
"Do school breakfast clubs work? We surveyed the principals of 150 schools that run a breakfast club, chosen at random from the national list of participating schools. 96 responded (64%). Of those, 88% reported that attendance had improved since the club began, and 71% reported improved concentration in class. On average, principals reported attendance up 4 percentage points. Breakfast clubs improve school attendance and concentration, and the programme should be funded in every school."
Evaluate this report. Make three points and give an overall judgement.
Point 1 — There is no comparison group
Identify. Every school surveyed runs a breakfast club. No schools without one were surveyed.
Justify. With no comparison, there is nothing to attribute the change to. School attendance moves year to year for many reasons — a mild winter with less illness, a change in truancy enforcement, wider economic conditions affecting family stability. A 4-point rise across schools generally would produce exactly this result with no effect from breakfast clubs at all. The design also cannot address regression to the mean: schools that started a breakfast club are likely to have done so after a period of poor attendance, and poor years are followed by better ones regardless of intervention.
Evaluate. The conclusion is causal — clubs "improve" attendance — but the study contains no element capable of supporting causation. Even taking every figure at face value, the report shows that attendance rose in schools with clubs, which is consistent with the clubs working, with a national trend, and with regression to the mean, and cannot distinguish between them.
Point 2 — The outcome is measured by the person being evaluated
Identify. All the outcomes are principals' self-reports, not measured attendance data.
Justify. Principals are being asked to assess a programme their own school chose to run, and which they will want funded. Three mechanisms push the same way:
- Social desirability and sponsorship effect — reporting failure to the programme's funder is unattractive.
- Confirmation bias — someone who believed the club would help notices the evidence that it did.
- Recall rather than measurement — "improved concentration" is an impression, not a variable. Even the 4-point attendance figure is reported by principals rather than extracted from roll data, so it carries the same bias.
Evaluate. This matters more than it would in most reports, because attendance is routinely and objectively recorded by every school in the country. The researchers had access to a hard measure and used a soft one instead. The 88% and 71% figures therefore measure principals' beliefs about the programme, which is a legitimate thing to know but is not what the conclusion claims to have established.
Point 3 — Non-response, and the direction it pushes
Identify. 96 of 150 responded — a 64% response rate, leaving 54 schools unaccounted for.
Justify. Non-response is unlikely to be random here. A principal whose breakfast club is running well and is proud of it has an incentive to reply; one whose club collapsed through lack of volunteers, or who saw no benefit, has less reason to spend time on a questionnaire about it. So the missing third is plausibly enriched with the least successful clubs.
Evaluate. 64% is a respectable response rate by survey standards, and it would be unfair to treat it as fatal — this is a genuine strength relative to many reports. But because the plausible direction of non-response bias is the same direction as the reporting bias in Point 2, the two compound rather than cancel, and the 88% figure should be read as an upper bound.