SETC-HOWREAD-0053 · How to Read · Class No.0053
English Reading and Comprehension · The Scientific Read · The Quantitative Read
A study can find that a lesson worked without proving that the lesson works for every learner, subject, teacher and year.
Education research sits inside unusually noisy systems. Students arrive with different prior knowledge. Teachers adapt. Classrooms interact. Assessments sample only part of learning. An intervention can improve immediate performance yet fail to transfer months later. A statistically detectable difference can be too small to matter educationally—or a modest average can hide large benefits for one group and none for another.
The Education Research Read teaches readers to follow the full claim chain: question → population → design → intervention → comparison → measure → analysis → result → uncertainty → classroom interpretation → transfer. The purpose is not to make teachers obey studies. It is to make evidence useful without making it larger than it is.
Scope. All classroom datasets and conversations below are original teaching examples. This article is not a clinical, psychological or special-education diagnosis and does not prescribe a universal teaching method. Research findings should be interpreted for the population, intervention, outcome and conditions actually studied. Maren, Iona and Leonie are fictional learning companions.
Question and claim · Research design · Assessment and measurement · Effect sizes · Classroom interpretation · Retention and transfer · Evidence synthesis · Kindergarten to JC · Casebook
1. Read the research question before the headline
“New method boosts learning” is not yet a research question.
Ask:
- Which learners?
- Learning what?
- Compared with what?
- For how long?
- Measured how?
- Under which teaching conditions?
A precise question might be: “Among Secondary 2 students who have already learned linear equations, does weekly cumulative retrieval practice improve performance on an unseen algebra test eight weeks later compared with otherwise similar weekly practice without cumulative retrieval?”
That sentence is less exciting than a headline. It is more useful because the learning job is visible.
2. Worked case: “reading every day makes children smarter”
Original finding. A survey finds that pupils who report reading more frequently also score higher on a vocabulary test.
Weak summary. “Reading every day makes children smarter.”
Several changes occurred. Association became causation. Vocabulary score became general intelligence. Self-reported frequency became a prescribed daily dose. The population disappeared.
A controlled summary is: “In the surveyed pupils, more frequent reported reading was associated with higher vocabulary scores.”
The next research question could test mechanisms. The original survey does not settle them.
3. Population controls the size of the claim
A study of forty university volunteers cannot automatically become a law about six-year-olds. A study in one language cannot automatically establish identical effects in another language. A study of expert learners can produce different results from novices because prior knowledge changes the task.
Read inclusion criteria, age, subject, prior attainment, language, setting and recruitment.
4. Sample and population are different nouns
The population is the group the study seeks to understand. The sample is the group actually observed.
A large sample can still be unrepresentative. A smaller carefully selected sample can answer a narrower question well.
5. Worked case: the voluntary revision app
A school invites students to try a revision app. The students who choose it already complete more homework and have higher prior scores. At term end they outperform non-users.
The result does not isolate the app effect. Self-selection and prior differences provide alternative explanations.
A useful sentence is: “App users achieved higher scores, but because use was voluntary and users differed beforehand, the comparison does not by itself establish that the app caused the difference.”
6. Research design determines which claims become possible
Different designs answer different questions.
- descriptive studies characterise;
- correlational studies examine relationships;
- randomised experiments strengthen causal inference under appropriate implementation;
- quasi-experiments use naturally occurring or designed comparisons without full random assignment;
- qualitative studies examine process, meaning and context;
- mixed-method studies combine evidence types.
Do not rank them on one ladder. Match design to question.
7. Random assignment is not random sampling
Random assignment distributes participants across conditions. Random sampling selects participants from a population.
A trial can randomise treatment perfectly inside one unusual school and still have limited generalisability beyond that school.
8. Worked case: excellent internal comparison, narrow external scope
One specialist mathematics school randomly assigns 120 high-attaining students to two feedback conditions.
The random assignment can support a strong causal comparison within the trial if implementation and attrition are sound.
It does not establish identical effects for all schools or attainment levels.
Internal validity and external validity answer different questions.
9. Control condition is not automatically “no teaching”
A comparison group may receive business-as-usual teaching, an alternative intervention, attention-matched activities or delayed treatment.
The effect means “difference relative to this comparison.”
If both groups receive strong teaching, a small difference can coexist with substantial learning in both.
10. Worked case: both groups improve
Group A rises from 50 to 72. Group B rises from 51 to 68.
Saying “Method B failed” because A ends higher ignores B’s improvement. The trial asks about the difference in gains under the design, not whether one group learned nothing.
Report both trajectories.
11. Baseline differences matter
If one group begins substantially ahead, end-point comparison alone can mislead.
Researchers may use baseline-adjusted models or analyse change, depending on design and assumptions.
Read the actual analytic strategy rather than inventing one from a graph.
12. Attrition can break an apparently fair comparison
Students leave studies, miss tests or fail to complete surveys.
If attrition differs by group or relates to outcome, the final observed sample can be biased.
13. Worked case: difficult programme, disappearing participants
Thirty percent of students leave Intervention A because the schedule is demanding; five percent leave Comparison B. Only completers are analysed.
The final A group may contain unusually persistent students.
A high completer score does not describe everyone originally assigned.
14. Intention-to-treat preserves assignment
In randomised trials, intention-to-treat analysis generally compares participants according to original assignment, helping preserve randomisation.
Per-protocol analysis asks a different question about those who adhered.
Both can be informative when labelled correctly.
15. Fidelity asks whether the intervention actually occurred
A study of “daily feedback” cannot test daily feedback if teachers delivered it twice a month.
Implementation evidence belongs beside outcome evidence.
16. Adaptation is not automatically failure
Teachers may need to adapt examples, pacing or language while preserving core mechanisms.
The key question is which components are essential and which are flexible.
17. Teacher effects can interact with intervention effects
A method may depend on training, subject knowledge or classroom management.
One unusually skilled teacher can make an intervention appear easier to scale than it is.
18. Cluster randomisation changes the unit of assignment
Schools or classes may be randomised rather than individual pupils.
Students within one class share teacher and environment, so observations are statistically clustered.
Analysis should account for that structure.
19. Worked case: 600 pupils but only 12 classes
A headline celebrates a sample of 600 pupils.
If the intervention was assigned to only twelve classes, the number of independent assignment units is much smaller than 600.
The design can still be valuable, but precision depends on clustering.
20. Quasi-experiments use structured comparison
Regression discontinuity, difference-in-differences and matched comparisons can support causal inference under assumptions.
They are not “almost experiments” in a casual sense. Each has a specific identification logic.
21. Difference-in-differences reads changes relative to changes
If School A adopts a policy and School B does not, researchers can compare how outcomes changed in A relative to change in B.
The method often relies on assumptions about counterfactual trends.
Read pre-treatment trajectories.
22. Regression discontinuity uses a threshold
If students just above and below an eligibility cut-off are otherwise comparable, a discontinuity can estimate a local treatment effect near that threshold.
Do not generalise the local estimate automatically to students far from the cut-off.
23. Qualitative research can explain mechanisms invisible in scores
Interviews and classroom observations can reveal how students interpret feedback, why teachers adapt a programme and where participation breaks down.
Qualitative depth does not provide population prevalence by itself.
24. Worked case: same score, different learning experience
Two classes show similar test gains. Interviews reveal that one class became more independent while the other relied heavily on teacher prompting.
The test result and interview evidence answer different questions.
A mature evaluation keeps both.
25. Assessment is a sample of performance
No test captures everything a learner knows.
Assessment selects content, response format, timing and scoring rules.
Therefore ask what construct the measure represents.
26. Reliability and validity differ
Reliability concerns consistency. Validity concerns whether interpretations and uses of scores are supported.
A test can reliably measure the wrong thing.
27. Worked case: reliable typing test used as writing measure
A timed computer task consistently rewards fast typing and sentence copying. Researchers call the score “writing quality.”
High reliability does not establish that the measure captures composition quality.
Construct alignment must be examined.
28. Near tests can favour the taught format
If the intervention practises the exact question type used in the outcome test, gains may reflect task-specific familiarity.
A far-transfer test asks whether learning survives a changed surface.
29. Worked case: memorised template
Students practise one essay scaffold and score higher on an assessment requiring the same scaffold.
A new prompt requiring different organisation reveals no advantage.
The intervention improved near performance but evidence for flexible writing transfer is weak.
30. Teacher-made tests can be valuable and vulnerable
They can align closely with curriculum and classroom goals.
If teachers know treatment condition, item selection or scoring can unintentionally favour one group.
Independent scoring or pre-specified rubrics can reduce some risks.
31. Standardised tests improve comparability but narrow the construct
Standardisation helps align administration and scoring.
It does not make a test a complete measure of learning.
32. Ceiling effects hide growth at the top
If high-performing students begin near maximum score, improvement may be impossible to observe.
A null result can reflect measurement ceiling.
33. Floor effects hide growth at the bottom
If tasks are too difficult, several learners can score near zero despite meaningful differences in partial understanding.
34. Worked case: both groups “unchanged” at 95%
Students begin at 95% and end at 96% and 97%.
The test provides little room to detect improvement.
Do not conclude the intervention cannot help advanced learners from this measure alone.
35. Rubrics contain constructs
A writing rubric may weight grammar, organisation, evidence and originality.
Changing weights changes the total score.
Read subscales before treating the total as one natural property.
36. Inter-rater reliability matters for judged work
Essays, presentations and projects require human judgement.
Training, moderation and blind scoring can improve consistency.
37. Blinding can reduce expectation effects
If scorers know which students received the intervention, expectations can influence subjective ratings.
Blind scoring is easier for some outcomes than others.
38. Self-report measures beliefs, not necessarily behaviour
“I study two hours daily” can differ from logged behaviour.
Self-report is valuable for attitudes and perceptions but should not automatically become objective behavioural data.
39. Attendance is not engagement
A student can attend physically and disengage cognitively.
Another can miss class but study independently.
Proxy measures need explicit limits.
40. Engagement itself is multidimensional
Behavioural, emotional and cognitive engagement can diverge.
A single “engagement score” may compress them.
41. Statistical significance asks a narrow question
A p-value does not tell us whether an effect is large, important, replicable or educationally worthwhile.