Bruno: Beware Of "Breakthrough" Education Research
Keith Humphreys had a fascinating post last week explaining why "breakthrough medical findings" - he uses the example of fish oil pills - often don't live up to the hype after additional research comes in.
He's talking about medical research, but I think the same issues arise in education research all the time. As Humphreys explains, it's difficult to perform a large-scale, well-controlled experiment to test out a new idea, and journals aren't interested in publishing small studies that find small (or no) effects. If you do enough small experiments, however, eventually you'll come up with large effects just by chance. Those results might be exciting enough to get published, but they may not be borne out by larger subsequent experiments.
If you follow education research at all, you know it's not uncommon for journals to publish studies with small sample sizes. That's often justifiable - it's hard to do big, well-controlled experiments to test educational interventions - but it does mean that big, novel findings from such studies should be taken with a grain of salt. Think of them as clues or stepping stones rather than reasons to dramatically rethink schooling. - PB (@MrPABruno) Image via Flcker CCommons