When research shows one training method works better than another, a recovery tool speeds the process of bouncing back from a hard workout, or a supplement boosts fitness or delays fatigue, athletes eager to optimize their performance take notice—and, often, action.
Many times, they do their due diligence too, seeking out the vetted, peer-reviewed papers that stand as the benchmarks of sound science. But not every published paper turns out to be true in the long run or the real world.
To some degree, that’s part of the nature of science. No study can perfectly reproduce what an athlete faces in training or competition. There’s always uncertainty involved in interpreting the results. A single blockbuster finding may make headlines—but it’s the accumulation of knowledge over time that points closer to the truth.
However, in many cases, exercise science has been tainted by what some experts have called questionable research practices (more on that, below). These methodological and statistical techniques often make research faster and easier to do, and study results more dramatic and clear-cut.
But they also dilute the quality and reliability of findings and tarnish the reputation of the field, said Aaron Caldwell, a Ph.D. candidate in exercise science at the University of Arkansas-Fayetteville, who also ran track and cross country as an undergrad at Baker University.
Join Runner's World+ for unlimited access to the best training tips for runners
And, they might lead athletes to waste time and money on things that just don’t work—or even worse, damage their health or their careers (for instance, when shoddy science is used to support questionable policiesabout who can compete).
“I think where the current status of research and sports science is—there’s some good stuff, and there’s some bad stuff,” Max Paquette, Ph.D., a biomechanist who studies runners and an associate professor in the School of Health Studies at The University of Memphis, told Runner’s World. “Many people don’t know how to decipher between good and bad stuff. As a result, it leads to confusion and frustration, and it doesn’t make sports science look good.”
Now, inspired by similar movements in psychology and biomedical science, a group of scientists is shining a spotlight on these troublesome techniques and calling for their colleagues to do better. As runners who train hard and aim to make the most of whatever measure of genetic talent we’re given, we’d be well served to join them.
What’s the big problem, anyway?
Back in December, 36 researchers—spearheaded by Caldwell and including well-known names in the strength-training world like Bret Contreras, Ph.D. C.S.C.S. and Brad Schoenfeld, Ph.D., C.S.C.S.—published an editorial on the open-access site SportRχiv, as first reported by Christie Aschwanden on FiveThirtyEight. In the editorial, they highlight some of the most common problems in exercise and sports science research and propose solutions. And they’ve launched a new organization—the Society for Transparency, Openness, and Replication in Kinesiology—to put their ideas into action.
You don’t need a Ph.D. to grasp how the issues they point out could sway science. (And they don’t even represent a comprehensive list of dubious tactics, said the corresponding author of the editorial, Northwestern University graduate student Andrew Vigotsky—there’s too much to fit into one paper.) The problems described include:
- Publication bias, sometimes referred to as the “file drawer problem.” Only studies deemed interesting, or those that show a given technique or product works (rather than a study showing something has no effect), tend to be submitted to journals for publication in the first place. The others never leave the file drawer or hard drive. Making it to print, especially in top-ranking journals, is even tougher. That means that if five experiments found a certain shoe or piece of gear didn’t improve performance, but one experiment suggested it might, the single study showing an effect may be the only one that winds up in the literature—providing a picture that’s incomplete and misleading.
- HARKing, or hypothesizing after the results are known, a.k.a. science in reverse. You start not knowing exactly what you’re looking for, collect a ton of data, then slice and dice it to see if there are any interesting patterns. Which is a fine way to generate questions for future study, Vigotsky said—but not to find evidence that a certain supplement helps you run faster or a lifting strategy builds bigger muscles. If you essentially come up with your questions after you already know the answers, it’s more likely you’ll overinterpret findings that were due to chance in the first place.
- P-hacking, which even many scientists can’t wrap their heads around.Essentially, it involves running multiple data analyses—many types of which are now available with the click of a mouse—until you get a result that appears statistically significant per a number called the p-value. You can also remove outlying results that don’t fit your existing narrative or break things down into subgroups. This is how, a few years ago, a devious journalist used real data to fool us with fake conclusions about chocolate and weight loss.
Though it’s not clear exactly how much sports research may be suspect, one study found widespread p-hacking throughout science. And another found inconsistencies in the way health and exercise scientists understand statistical significance.
Why is this happening in the exercise field?
While fraud and intentional misconduct surely exist, most of these missteps likely aren’t due to malice, Caldwell said. Many researchers might not be attuned to the nuances of different statistical techniques, nor are they always well educated in proper methodology.
The culture of science offers ample incentives to cut corners. Academic jobs depend on publishing frequently. “People tend to judge quantity over quality, because it’s so difficult to judge quality,” Vigotsky said. It’s human nature to want to craft a good story—and to read one, if you’re a peer reviewer advising an editor at a journal on which papers to accept and which to reject.
Many exercise science studies are funded by a company testing a product. That doesn’t mean they’re inherently biased or should be dismissed outright. But it does mean there’s even less of a chance the results will find their way into a journal if they don’t show an obvious benefit to that company’s product or service.
Without conscious intervention to counter these forces, it’s possible exercise science could face a takedown along the lines of what happened to food behavior researcher Brian Wansink. For years, his studies about how things like buffet prices, bowl sizes, and food labels affect how much you eat grabbed headlines and shaped policies.
Last year, journalists and colleagues began questioning his methods. He’s now had a total of 15 journal articles retracted, and resigned after an investigation by Cornell University found he committed academic misconduct by using similar tactics to manipulate data.
The prospect of a similar shakeup in exercise science “keeps me up at night,” Caldwell said. “I think there’s a lot of skeletons in the closet already,” though he’s not sure exactly which lab doors to look behind.
So what’s the answer?
The editorial proposes a number of specific methods to make the research process more transparent and less vulnerable to misuse or downright abuse, including:
- Classifying studies differently depending on whether the experiment is generating hypotheses or testing one in particular. These categories—exploratory and confirmatory —would help signal to readers whether a finding was early and speculative or more likely to hold up over time.
- Pre-registering study plans in advance, at a repository like those hosted by the Center for Open Science or clinicaltrials.gov. That way, researchers would have to think through their study design carefully before they ever collect a single point of data—and others would be able to tell if they changed their analysis midstream.
- Using a publication format called a registered report, at least for some studies. In this model, a research plan goes through the process of peer review before the experiment begins. Fellow scientists can provide comments on how the study is set up, and the statistical techniques planned to analyze the data, before it’s too late to make changes. Once they approve a plan, provided it’s followed, the results would also be accepted for publication in the same journal. Not every journal accepts registered reports—but an increasing number will consider them, and STORK plans to launch its own journal soon as well.
Paquette, who’s not an author on the editorial, said he has never submitted a registered report, but would be open to the idea, especially for long and involved studies. And he takes other steps personally to address these issues: For instance, if his lab accepts grant money from a company, he includes a line in the contract stipulating that he’ll publish the results regardless of the outcome—meaning even if they don’t show the product being studied works.
How can runners make the most of the training science out there?
Both broadly and for individual scientists, these solutions face obstacles and will take time to implement. “Science is hard,” Vigotsky said—and we get it. But so is that 10th 200-meter repeat or the final 0.2 of a marathon. That’s no excuse for not researchers not giving their best effort, especially when the results have real-world implications for athletes like us.
In the meantime, we can all work to increase our scientific literacy. Even just understanding the process of science—and how much uncertainty is involved—goes a long way in avoiding being duped.
Be on the lookout for results that seem too good to be true, and research articles that don’t supply hard numbers but just claim a result is “significant.” Also, be wary of results published by those pushing an agenda, such as a specific diet, who might have more reason to cherry-pick their results, Paquette said. (Another scientist and co-author on the editorial, Greg Nuckols, has even more advice here.)
If you want to take things a step further, ask an exercise scientist for help interpreting research you’re interested in—or follow along with their chatter on Twitter, where they’re always debating methods and ideas (Caldwell is at @ExPhysStudent, Vigotsky at @avigotsky, and Paquette at @BiomechMax). You can even go full-on science geek and join STORK. You don’t have to be a researcher to support the organization on Patreon, and membership dues begin at an affordable $3 a month for non-academics.
“Exercise and sports science has a long way to go. We're young, so please forgive us if you feel like we've betrayed your trust at any point,” Caldwell said. “We’re still learning about how to do the best science, as all science is constantly doing.”
Cindy is a freelance health and fitness writer, author, and podcaster who’s contributed regularly to Runner’s World since 2013. She’s the coauthor of both Breakthrough Women’s Running: Dream Big and Train Smart and Rebound: Train Your Mind to Bounce Back Stronger from Sports Injuries, a book about the psychology of sports injury from Bloomsbury Sport. Cindy specializes in covering injury prevention and recovery, everyday athletes accomplishing extraordinary things, and the active community in her beloved Chicago, where winter forges deep bonds between those brave enough to train through it.
