Evaluating your sources and the evidence
Why the source of a claim matters
The Excellence criterion for this standard names commenting on sources and information as one way to analyse and prioritise physics knowledge. It lists three things to consider: validity, bias, and how science ideas are used by different groups.
Validity: can this information be relied on?
- Date. Is it current? A 2011 cost figure for solar panels is worthless today, because module prices have fallen by around 90% since. Physics constants do not go stale; measurements, costs and capabilities do.
- Peer review. Has the work been checked by other scientists before publication? A peer-reviewed paper has cleared a bar that a blog post, a press release or a YouTube video has not.
- Scientific acceptance. Is this the mainstream position or an outlier? A single paper contradicting a large body of established work is not equivalent to that body of work, however striking its claim.
- Primary or secondary? Is it the original measurement, or someone's summary of it? Each retelling can drop the uncertainty, the conditions or the sample size that made the original meaningful.
Bias: whose interests does this serve?
- Bias is not the same as dishonesty. A source can be entirely factual and still be selective about which facts it presents.
- Ask: who produced this, who paid for it, and what would they like the reader to conclude?
| Source | Likely to be reliable for | Treat with care when it says |
|---|---|---|
| Manufacturer datasheet | technical specifications | how it compares with rivals; expected lifetime |
| Industry association | industry-wide statistics | the size of the benefits, the size of the risks |
| Campaign group | identifying who is affected | quantitative risk figures |
| Government agency (e.g. MBIE, EECA, GNS Science) | measured national data | policy already committed to |
| Peer-reviewed journal | the specific measurement reported | anything outside the paper's own scope |
How different groups use the same science
- A useful analysis often finds that two sides agree on the physics and disagree about something else. Saying so is a strong Excellence move, because it identifies precisely where the dispute really lies.
- On wind farms, both sides accept and the resulting intermittency; they disagree about how much value to place on visual amenity and on firming capacity.
- Watch for quantities used without their scale. "Radiation was detected" is meaningless without a dose; "emissions were reduced" is meaningless without a baseline.
- Watch for a correct number used to support a conclusion it does not support — a real measurement, quoted accurately, but about a different situation.
Judging a risk claim with physics
Most socio-scientific issues involve a risk claim, and physics gives you a few reliable tests.
- Is the mechanism plausible? For radiation, compare the photon energy with the energy needed to ionise an atom (a few eV). Radio and microwave photons are thousands of times too weak, so the DNA-damage mechanism that applies to X-rays does not apply to them.
- What is the scale? Compare the quantity in question with a familiar benchmark — natural background radiation, the energy in a household appliance, the power of the Sun on a square metre.
- How does it vary with distance? Anything radiating from a point obeys an inverse-square law, so doubling the distance quarters the intensity. That single fact settles many proximity arguments.
Writing a source comment that earns credit
Three parts: what the source is, what you rely on it for, and what you check elsewhere.
- Weak: "This website may be biased."
- Strong: "The turbine manufacturer's brochure is a reliable source for the rated power and rotor diameter, since those are specifications they must meet contractually. Its stated capacity factor of 45% is a figure I have checked against the Electricity Authority's published generation data for existing New Zealand wind farms, which gives 35–40% — the manufacturer's figure is likely to describe an ideal site rather than a typical one."
Worked ExampleEvaluating three sources on one issue
A student researching whether a proposed 5G transmitter near a school poses a health risk finds: (1) a peer-reviewed review of radio-frequency exposure studies from 2023, (2) a telecommunications company's information page, and (3) a community campaign group's flyer citing "a 2015 study showing DNA damage from mobile phone radiation". Evaluate each and state what you would take from it.
Step 1 — The peer-reviewed review (2023)
Validity: high — recent, peer reviewed, and a review rather than a single study, so it summarises the weight of evidence rather than one result. Bias: low, though the funding source should still be checked. Use it for: the mainstream scientific position and the exposure limits, which is the strongest evidence available to me.
Step 2 — The telecommunications company's page
Validity: the technical specifications — transmitter power, frequency, distance to the nearest classroom — are reliable, because they are regulated figures the company must meet. Bias: the company has a direct commercial interest in the installation proceeding, so its framing of the risk is not independent. Use it for: the numbers I need for a calculation. Check elsewhere: any claim that exposure is "completely safe", which is a conclusion rather than a measurement.
Step 3 — The campaign flyer
Validity: the cited study is from 2015, before 5G deployment, so it does not concern the technology in question. A single study is also not equivalent to a review, and the flyer does not say whether the result has been replicated — replication is what turns a single finding into evidence. Bias: the group formed to oppose this installation, so it will select findings that support that position. Use it for: identifying who is concerned and why, which is a genuine social factor and belongs in the response. Do not use it for: a quantitative risk claim.
Step 4 — Apply the physics test to the underlying claim
5G operates at frequencies up to about GHz in New Zealand. The photon energy is
Ionising an atom takes of the order of eV — around a million times more. So the mechanism proposed (direct DNA damage, as caused by X-rays) is not physically available at these frequencies, whatever a single study reported.
Step 5 — State what the evaluation contributes
The physics does not merely favour one source over another; it explains why the sources disagree. The review and the physics agree because both are constrained by the photon energy; the flyer's claim rests on a mechanism the physics rules out. That is a source comment doing real work — it uses physics to prioritise the evidence, which is exactly what the Excellence criterion asks for.