Bruno: Beware Of "Breakthrough" Education Research

Keith Humphreys had a fascinating post last week explaining why "breakthrough medical findings" - he uses the example of fish oil pills - often don't live up to the hype after additional research comes in.

He's talking about medical research, but I think the same issues arise in education research all the time. As Humphreys explains, it's difficult to perform a large-scale, well-controlled experiment to test out a new idea, and journals aren't interested in publishing small studies that find small (or no) effects. If you do enough small experiments, however, eventually you'll come up with large effects just by chance. Those results might be exciting enough to get published, but they may not be borne out by larger subsequent experiments.

If you follow education research at all, you know it's not uncommon for journals to publish studies with small sample sizes. That's often justifiable - it's hard to do big, well-controlled experiments to test educational interventions - but it does mean that big, novel findings from such studies should be taken with a grain of salt. Think of them as clues or stepping stones rather than reasons to dramatically rethink schooling. - PB (@MrPABruno) Image via Flcker CCommons

Comments

[dan]

Agreed! I talked about this at some length in my latest book.

[Dan]

Agreed! I wrote about this at some length in my recent book.

[Paul Bruno]

@Dan - I have a copy of your book sitting on my desk as we speak. I am embarrassed to say I haven't read it yet, however. It's next on my list, as soon as I'm finished with "On Being Certain".

[Jessica]

@Paul-Very good point. My county loves to look at and analyze data. But the problem we often run into is unknown variables; especially, variables that have to do with students' lives outside of school. I think education researchers would have similar problems controlling all the variables.

@Dan-I'll have to check out that book!