The first time we dip our toes into the freezing water of research, we realize there is a “unique” language that takes a while to learn. For example, we learn about statistical tools that sound like a foreign language, biases that come from unexpected directions, and of course, the concept of statistical significance and its critical value to any quantitative study. To the novice researcher, achieving statistical significance is paramount to show the outcomes were not a result of chance. (There is even a word for the incessant focus on producing statistically significant results: “significosis.”)1
But does that mean readers can accept the findings of the study and use them in clinical decision making? Perhaps not as much as readers might think. In the beginning of developing a research project, the researcher does all he/she can do to avoid biases that might affect the outcome. But there are two types of bias that, albeit similar, demand quiet, reflective thinking – publisher bias and outcome reporting bias. Studies with significance in results are more likely to be published than those with non-significance.2 As a matter of fact, a recent study by Dickerson, Chan, Chalmers, Sacks, and Smith Jr. suggests that “papers with statistically significant results are three times more likely to be published than those with null results.”3 Moreover, published papers may selectively report outcomes favoring those that are statistically significant.4
Publisher bias occurs when publishers and their editorial team are less likely to publish a study because the results do not reach statistical significance, even when the study is well designed. It has been suggested that competition for citation index and financial survival of journals makes it more attractive to publish positive findings.5 Null results don’t attract positive media attention, don’t persuade investors, and it simply doesn’t feel good. But publishing only results with significant findings upsets the balance of findings in favor of positive results, distorting what we think we know about a specific topic.5
Authors are also guilty of “outcome reporting bias” when they decide not to submit an article for publication because no significant difference was found. The highly competitive environment for funding and the drive for career advancement likely incites researchers to submit only studies with significance for publication, knowing that they are more likely to be considered for publication by editors, more favorably reviewed by peers and, once published, more likely to be cited.5 In fact, trials funded by industry have been shown to be twice as likely to result in non-publication compared to trials sponsored by academia despite results being posted in a trial repository.6
In the case of clinical trials, withholding negative results from publication — publication or outcome reporting bias — could have major consequences for the health of millions. In preclinical and experimental research, this bias may seriously distort the literature, drain scarce resources by undertaking research in futile quests, and lead to misguided research and teaching practices. Over and above scientific considerations, research participants consent to participate in research on the understanding that they are contributing to advances in treatment and scientific knowledge. Our ethical duty as researchers and editors is to honour this engagement and publish both positive and negative outcomes in an equitable manner.5
In health care, it is vitally important that we know what works and doesn’t work in the treatment of our patients. Take ultrasound, for example. In the 1960’s until the 2000s, one of the most common treatments for musculoskeletal pain was ultrasound. “Ultrasound” was almost a household name. It wasn’t until the 2000s when the Philadelphia Panel studied ultrasound as a treatment for acute low back pain, sub-acute low back pain, post-surgical back pain, chronic low back pain, chronic neck pain, knee osteoarthritis, and calcific tendinitis of the shoulder. The results showed that ultrasound was only effective for calcific tendinitis of the shoulder.7 Is it possible that there were studies that showed limited effects of ultrasound and no significance for the other conditions but were withheld from publication for 30 to 40 years?
Well-designed studies should be published whether or not statistical significance is found. After all, the absence of “significance” when significance is expected is significant in itself.
- Antonakis, J. (blog). Got “significosis?” Here are the five diseases of academic publishing. Retraction Watch https://retractionwatch.com/2017/02/21/got-significosis-five-diseases-academic-publishing/
- Song F, Parekh S, Hooper L, Loke YK, Ryder J. Dissemination and publication of research findings : an updated review of related biases. Health Technol Assess 2010;14(8)
- Dickersin K, Chan S, Chalmers TC, Sacks HS, Smith H Jr. Publication bias and clinical trials. Control Clin Trials. 1987;8(4):343-353. doi:10.1016/0197-2456(87)90155-3
- Chan AW, Hro´bjartsson A, Haahr MT, Gøtzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA. 2004; 291(20):2457-2465.
- Joober R, Schmitz N, Annable L, Boksa P. Publication bias: what are the challenges and can they be overcome?. J Psychiatry Neurosci. 2012;37(3):149-152. doi:10.1503/jpn.120065
- Pica N, Bourgeois F. Discontinuation and Nonpublication of Randomized Clinical Trials Conducted in Children. Pediatrics . 2016 Sep;138(3):e20160223. doi: 10.1542/peds.2016-0223.
- Čota S, Delimar V, Žagar I, et al. Efficacy of therapeutic ultrasound in the treatment of chronic calcific shoulder tendinitis: a randomized trial. Eur J Phys Rehabil Med. 2023;59(1):75-84. doi:10.23736/S1973-9087.22.07715-2
Recommended Citation
Hill CJ, Kolber MJ. The Significance of Significance. The Internet Journal of Allied Health Sciences and Practice. 2024 Jun 26;22(3), Article 10.
