Organising the evidence and evaluating the process
What "organising" means here
- Step six of the process is organising the evidence, and it is assessed as part of following the process.
- Organised means arranged so that the evidence answers the focusing questions, not arranged chronologically or by source type.
- By question — each focusing question with the evidence bearing on it. This is almost always the right arrangement, because it makes relevance visible.
- By theme within a question, where a question has parts.
- What organisation is not: a list of sources in the order you found them, or a folder sorted by primary and secondary.
- The test: can an assessor see, without asking, which evidence answers which question? If not, the evidence is collected rather than organised.
Evaluating the process — what Achieved requires
- The standard names three things, and each needs its own material:
- The successes and difficulties encountered in conducting the research.
- How the line of inquiry may have changed as evidence was accumulated.
- The issues to consider for future inquiries.
- The middle one is the one that must be recorded as you go. You cannot describe a change of direction you did not notice at the time.
- "Difficulties" does not mean I ran out of time. It means: this collection was not indexed; this source contradicted that one and I had to decide which to prefer; this question turned out to be unanswerable from what survives.
Evaluating the process — what Excellence requires
- The descriptor has three clauses, and they escalate:
- analysing the strengths and weaknesses of the process
- analysing how these are likely to impact on the VALIDITY OF THE FINDINGS
- considering alternative research steps and/or lines of inquiry and/or methods, and their implications
- The second clause is what makes this Excellence rather than Achieved. Achieved describes the process; Excellence argues about how much the conclusions can be trusted.
- The third clause is a counterfactual — what a different approach would have produced. It is argued from the character of the sources, since no evidence bears on it directly.
The four things to record while you work
- Each maps onto a clause of a descriptor, and none can be reconstructed afterwards.
| Record this | It evidences |
|---|---|
| The questions you abandoned, and why | How the line of inquiry changed |
| The sources you could not obtain, and what you think they held | Weaknesses, and their impact on validity |
| The searches that returned nothing | Strengths and weaknesses of the process — a null result is a finding about the record |
| Where you stopped looking, and why | Alternative research steps and their implications |
- All four are about absences, and absences are invisible unless written down at the moment they occur.
Modesty is not criticality
- Modest: I did not have enough time, and my findings might be incomplete. This says nothing an assessor could not have guessed.
- Critical: name a specific weakness, state the mechanism, and bound a specific finding.
- My account of the objections rests almost entirely on the newspaper record, because the submissions do not survive. The paper selected which objections to report, so my finding that they were mainly about cost may reflect what was thought newsworthy rather than what was said.
- That is one sentence and it contains all three Excellence clauses: a weakness, its mechanism, and its effect on a finding.
Worked Example
Worked example
Note: the Riversdale Flats Drainage Scheme is an INVENTED case.
Two students hand in evaluations. Both did similar research. Which reaches Excellence, and why?
Student A:
Student B:
Answer:
Step 1 — check Student A against the Achieved descriptor.
- Successes and difficulties? Named, but generically. Time is a difficulty every project has.
- How the line of inquiry changed? Not addressed at all.
- Issues for future inquiries? Start earlier and be more organised — advice about conduct, not about inquiry.
Student A has not fully met Achieved, because the middle clause is missing entirely.
Step 2 — check Student B against the three Excellence clauses.
- Strengths and weaknesses of the process? Yes, and both are specific to this project — a complete minute book, which is a strength, and its silence on discussion, which is the weakness that follows from the same fact. Noticing that a strength and a weakness are the same feature is analysis.
- Impact on the validity of the findings? Yes, and precisely. The finding is not abandoned; it is restated at the strength the evidence supports — the recorded ground for dismissal was cost. That reframing is the whole of the second clause.
- Alternative steps and their implications? Yes, with the trade-off stated in both directions — recollections are less precise on dates and are the only objector evidence available — and a reason for preferring it grounded in the skew already identified.
Step 3 — note the extra move that lifts it further.
Student B's third paragraph records a null search result and then does something unusual: refuses to resolve it. Either none were made in writing, or they were not filed — I cannot distinguish between those, and the difference matters. Stating what the evidence cannot settle, and why the undecided question matters, is a stronger position than choosing an answer the evidence does not support.
Step 4 — note where Student B's material came from.
None of it could have been written at the end. The failed search, the observation about advertising, the naming convention in the report — all were noticed while working and recorded at the time. Student A's evaluation is what remains when nothing was recorded: an accurate account of how the project felt.