Education research rarely rewards failed or inconclusive studies. Research that doesn’t turn out as planned simply never sees the light of day. But does that mean there are no lessons to been gleaned from them? That question is at the heart of School of Education Professor Hunter Gehlbach’s focus on open science — a new approach to scientific process and publication that emphasizes transparency in addition to significance of results.
He says that the journals’ desire to publish only studies with “significant” outcomes compromises the scientific process, often encouraging researchers to “torture data until it tells you what you want to hear.” But, Gehlbach says, in most cases a failed or inconclusive study can be just as valuable as one showing positive results. That is, demonstrating what doesn’t work is just as important as proving what does. The consequence of this overemphasis on “significance” is education policies based on flawed science. Usually, it’s the learners who suffer. Science’s responsibility is to reflect the real world.
The remedy is transparency. Gehlbach and colleagues are leading a field-wide charge to bring open science to education research. They have published guidelines to help researchers and publishers bring greater transparency to the process. Recommendations include public preregistration of methods and hypotheses, pre-print of working drafts, publication of rejected papers—and, yes, publication of null results—as well as the rewarding of replication studies to verify outcomes.
Open data. Open code. Open materials. Open science. That’s the bold idea, Gehlbach says. Educators, administrators — and learners — everywhere stand to gain.
The desire to publish what’s ‘sexy’ is too strong. The journals want to publish, and the researchers want data that is publishable. It leads to the publication of ‘illusory results’.
