The Next Time You Hear Some Edubabble Starting With The Words "Research Shows..."
..You can cut the person off by asking if they're sure the "research" they're citing isn't horseshit:
A painstaking yearslong effort to reproduce 100 studies published in three leading psychology journals has found that more than half of the findings did not hold up when retested. The analysis was done by research psychologists, many of whom volunteered their time to double-check what they considered important work. Their conclusions, reported Thursday in the journal Science, have confirmed the worst fears of scientists who have long worried that the field needed a strong correction.
The vetted studies were considered part of the core knowledge by which scientists understand the dynamics of personality, relationships, learning and memory. Therapists and educators rely on such findings to help guide decisions, and the fact that so many of the studies were called into question could sow doubt in the scientific underpinnings of their work.
...
More than 60 of the studies did not hold up. The project began in 2011, when a University of Virginia psychologist decided to find out whether suspect science was a widespread problem. He and his team recruited more than 250 researchers, identified the 100 studies published in 2008, and rigorously redid the experiments in close collaboration with the original authors.
Only 36% of the studies held up to scrutiny. If you add in the so-called research in other fields, even that kind of accuracy is probably too high:
Dr. John Ioannidis of Stanford, who predicted in the early 2000s that about half of the findings across medicine were inflated or wrong, said that the 36 percent replication rate for psychology might even be high once the results for other disciplines, like economics, animal research, cell biology and other areas of biomedicine, were also tested.“Many of the biases found in psychology are pervasive,” Dr. Ioannidis said.
It would be interesting to see a painstaking yearslong effort to reproduce 100 studies of education research to see what the replication rate is.
You hear people on all sides of education reform issues throw out the words "Research shows..."
Our culture has fetishized "research" and "science," as if to say if something has been found by "research" in a "scientific study," it must be so.
This is not so, as one commenter at the NY Times notes:
I applaud the researchers ability to recognize the limitations of their research and their standard of peer review. There is an old saying in medical research: "half of what is published is Perdido Street School: The Next Time You Hear Some Edubabble Starting With The Words "Research Shows...":
