How current research practices are damaging science, and how open science offers some solutions
Dorothy V M Bishop (University of Oxford)
Recorded: University of Leuven - 3 May 2021
N.B. A 35 min version of this talk is available, entitled "Productivity and reproducibility: conflicting pressures on scientists?" - • Productivity and reproducibility: conflict...
Summary
Quasi-cumulative science - where we think we are building on foundations of prior work, only to find the earlier work isn't solid - has a detrimental effect on science. There is no single cause for this, but rather an unfortunate combination of factors. Five of these, together with possible solutions, are discussed:
1. Publication bias - One solution is Registered Reports, which decouples decisions about publication from knowledge of research results.
2. Citation bias. Creates the impression of widespread agreement on a topic, because literature that does not support it gets forgotten - Systematic reviews are one method that attempts to redress the imbalance in citations, but it is not infallible. Unlikely to change unless we increase awareness of the serious consequences of citation bias, and train researchers to seek out and evaluate evidence contrary to their position.
3. P-hacking, or selective reporting of only positive findings from within a study, often after exploration of many different analysis options. I discuss one form of p-hacking which might be termed 'moving goalposts', where studies that appear to replicate an initial finding actually fail to do so, but instead present results on a related question - Registered reports offer a solution to p-hacking.
4. Low statistical power, often due to small sample sizes - Training scientists to explore simulated data is one way to counteract underpowered studies.
5. Obsession of funders/journals with novelty. A 'top down' influence that produces a distorted incentive structure for scientists, who avoid slow, careful science that builds up knowledge on a topic, and rather feel they must overhype results and jump from one hot topic to another - The solution is for funders and institutions to change how they evaluate researchers, to focus on rewarding those who adopt open, reproducible methods. The Hong Kong Principles for assessing researchers are introduced as one such approach.
Other resources:
https://www2.slideshare.net/deevybishop
http://deevybee.blogspot.com/2012/11/...
References
Bishop, D. (2019). Rein in the four horsemen of irreproducibility. Nature, 568(7753), 435–435. https://doi.org/10.1038/d41586-019-01...
Bishop, D. (2020). How scientists can stop fooling themselves over statistics. Nature, 584(7819), 9–9. https://doi.org/10.1038/d41586-020-02...
Border, R., Johnson, E. C., Evans, L. M., Smolen, A., Berley, N., Sullivan, P. F., & Keller, M. C. (2019). No support for historical candidate gene or candidate gene-by-interaction hypotheses for major depression across multiple large samples. American Journal of Psychiatry, 176(5), 376–387. https://doi.org/10.1176/appi.ajp.2018...
Chambers, C. D., Dienes, Z., McIntosh, R. D., Rotshtein, P., & Willmes, K. (2015). Registered Reports: Realigning incentives in scientific publishing. Cortex, 66, A1–A2. https://doi.org/10.1016/j.cortex.2015...
De Vries, Y. A., et al (2018). The cumulative effect of reporting and citation biases on the apparent efficacy of treatments: the case of depression. Psychological Medicine 48, 2453-2455. doi:10.1017/S0033291718001873
de Groot, A. D. (2014). The meaning of “significance” for different types of research [translated and annotated by Eric-Jan Wagenmakers, Denny Borsboom, Josine Verhagen, Rogier Kievit, Marjan Bakker, Angelique Cramer, Dora Matzke, Don Mellenbergh, and Han L. J. van der Maas]. Acta Psychologica, 148, 188–194. https://doi.org/10.1016/j.actpsy.2014...
Greenberg, S. A. (2009). How citation distortions create unfounded authority: Analysis of a citation network. BMJ, 339, b2680. https://doi.org/10.1136/bmj.b2680
Greenwald, A. G. (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82(1), 1–20. https://doi.org/10.1037/h0076157
Locascio, J. J. (2017). Results blind science publishing. Basic and Applied Social Psychology, 39(5), 239–246. https://doi.org/10.1080/01973533.2017...
Moher, D., Bouter, L., Kleinert, S., Glasziou, P., Sham, M. H., Barbour, V., Coriat, A.-M., Foeger, N., & Dirnagl, U. (2020). The Hong Kong Principles for assessing researchers: Fostering research integrity. PLOS Biology, 18(7), e3000737. https://doi.org/10.1371/journal.pbio....
Nissen, S. B., Magidson, T., Gross, K., & Bergstrom, C. T. (2016). Publication bias and the canonization of false facts. ELife, 5, e21451. https://doi.org/10.7554/eLife.21451
Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638–641. https://doi.org/10.1037/0033-2909.86....