The rush to publish and report during the pandemic is compromising quality, worried experts tell Stephen Armstrong from the British Medical Journal.
On 11 April Neel Shah, assistant professor of obstetrics, gynaecology, and reproductive biology at Harvard Medical School, published a grim assessment of the scientific research into covid-19 and its effects on pregnancy.1
“I’ve never felt as dependent as I am today on shaky data to make what could be life or death decisions,” he wrote. “In a normal month I . . . quickly cast aside studies that include just a handful of patients or provide no formal way of accounting for context. Yet today these kinds of studies are all I have to go on.”
Shah explains his concerns to The BMJ: “I understand the challenge of providing evidence based research [on the pandemic]. But people like me on the front line have to make life or death decisions based on the information that we have. We have to be willing to update what we believe more rapidly—and yet there’s so much information that is hard to trust it makes our jobs very difficult.”
Covid-19
The covid-19 pandemic has created an urgent need for scientific evidence to help politicians, doctors, researchers, and the general public understand this evolving situation. The problem is that good science, which requires scrutiny and replication, simply cannot move at the speed of the rolling news cycle. Over the past 20 years responses to the misreporting of medical theories has resulted in a series of checks and balances to protect all concerned from hasty or even bad science. The professionals at the helm of those controls, they tell The BMJ, are worried: quality seems to be slipping, and there are question marks over findings and problems with publishing and reporting.
System breakdown
Fiona Fox is chief executive at the UK’s Science Media Centre, established in the wake of the measles, mumps, and rubella vaccine controversy to ensure that the media covered scientific issues with the best evidence and expertise. “What we are seeing is worrying signs of a compromise in quality,” she says.
Many of the science stories covered in the news media come from press releases issued by around 10-15 of the world’s top journals, including The BMJ, the Lancet, and JAMA. The Science Media Centre gets sight of these before the embargos lift, allowing Fox’s team to recruit experts who can contextualise and explain the significance of the study to journalists. “When the deluge of new [covid] findings came out this embargo system broke down,” Fox explains. “Journals are now often releasing papers for immediate release, making it harder for us to get third party comments to the journalists in time.
“Suddenly all our tried and tested ways of helping journalists to report findings more responsibly have been removed. It’s making it hard for us to do our job, but it feels more important than ever. We don’t need to slow the wheels of science when people are dying, but neither do we need bad science that falls below acceptable standards and makes things worse.”
Moreover, Fox says, with the thirst for information on covid-19 far outstripping the usual demand for medical science coverage, journalists are reporting on more preprint studies that have not been peer reviewed and vary in quality.
The centre has recently issued retroactive contextualising—and in some cases critical—comments on several preprints, including one by a professor of microengineering that compared deaths from covid-19 and social distancing measures in nine countries,23 an observational study of covid-19, high blood pressure, and blood pressure lowering drugs,45 and a study that compared the risk of infection in different blood groups.67
Cite-bait and hype
Why, amid a global pandemic, should the quality of medical research and its reporting seem to drop suddenly? “Because there are imperfect incentives at every stage in the process,” says Marcus Munafò, professor of biological psychology at Bristol University’s MRC Integrative Epidemiology Unit, who leads the UK Reproducibility Network (UKRN), set up last year to improve research quality. “Part of what may be drawing researchers to this is seeing the amount of covid-19 work in respectable journals and thinking they can get published in those journals too.
“There is a similar risk for journals. Their incentive is to put something out there that’s ‘cite-bait’: journals survive by publishing stuff that people read and cite so that more people read the journal. Do journals like The BMJ have an expedited review process, and what checks and balances are in place?”
Munafò also points out, “Much of the hype in medical reporting can come from the press release—written by the university’s PR department but approved by the study authors. Then it reaches journalists who have a need for clicks or sales. All of those things have always happened, but they seem to be happening to a greater degree in the current climate.”
He believes most research is well intentioned but points to the UKRN philosophy: “Fast, cheap, good . . . you can only have two.” He adds, “There is an urgent need for data and knowledge, but false information is worse than no information.”
Speed versus quality
The issue was recently illustrated by one 22 April study in JAMA,8 which reported 88% mortality among patients ventilated for covid-19. Crucially, however, that figure did not include the relatively large number of patients still alive and ventilated at the time of reporting, but the paper was nevertheless covered in the media under headlines such as “New study shows nearly 9 in 10 covid-19 patients on ventilators don’t make it.”9
The authors have since issued a correction.10 This said that 38 ventilated patients were discharged alive, 282 died, and 831 remained in hospital, giving a ventilator mortality of 24.5% rather than close to 90%.
Karina Davidson, senior vice president of research for Northwell Health, the US chain of non-profit hospitals where the research was conducted, explained, “What we thought was important, given that we had so many patients who were presenting for hospitalisation over four days, was to get out their presenting symptoms, triage lab values, and initial course, so that we could have some descriptive data for others from our country.
“We usually report when total outcome denominators are known. In our article we reported only on the subset for which outcomes were known in those few days. Everyone is struggling with the correct balance between judicious careful processing of and great thirst for information on this brand new disease.”
Primary research and peer review
Malcolm Macleod, professor of neurology and translational neuroscience at the University of Edinburgh and its academic lead for research integrity and improvement, has no interest in identifying individual papers for criticism. “The improvements we need would be better served if every manuscript could be just a little bit better,” he explains.
Macleod is part of a team that has been working to categorise a total of more than 12 000 unique studies gathered internationally since the start of the outbreak. They have so far categorised 2181 publications, including 304 primary research papers, meaning that roughly 14% of publications include primary research. The proportion of primary research papers that have been peer reviewed is 27%.
“Seventy two per cent of the primary research studies have been observational stuff: ‘This is what it looks like in five patients of mine,’” Macleod explains. “Hardly any of those are peer reviewed or preregistered, so there is no protection against bias.
“There are a lot of journals that do not even say if the work has been peer reviewed, so they are presumably arrogant enough to believe we assume their papers have been peer reviewed, and yet there are some that have been accepted and published on the same day. That’s an issue for the journals going forward—they need to be more transparent.”
He accepts that the scientific and medical profession should hold research on covid-19 in the middle of this pandemic to a slightly different standard from other research—accepting a lower standard of evidence to take a drug through to clinical trial, for instance—but researchers need to go through the process of due diligence to be sure of the level of this standard.
“We think what improves quality is transparency at every step: sharing methodology, data, materials, code . . . everything,” says Munafò. “[Neil Ferguson’s] Imperial group’s original paper on the lockdown went up as a preprint,11 but they weren’t able to share the computer code they used to make their calculations at the same time because it was old code and hadn’t been prepared for sharing. This meant researchers couldn’t check their code.
“Lockdown is causing harm to human lives. Currently the view is that the benefits outweigh the costs. We need the decisions to be informed by high quality evidence, even if it is imperfect or incomplete. Transparency will help ensure the process is seen as trustworthy.”
Footnotes
- Provenance and peer review: Commissioned; not externally peer reviewed.
- Competing interests: I have read and understood BMJ’s policy on declaration of interests and have no relevant interests to declare.
References
- ↵
- Shah N
- ↵Pike WT, Saini V. An international comparison of the second derivative of COVID-19 deaths after implementation of social distancing measures. medRxiv [preprint]. doi:10.1101/2020.03.25.20041475.Abstract/FREE Full TextGoogle Scholar
- ↵Expert reaction to unpublished preprint paper on modelling deaths from COVID-19 after implementing social distancing measures in different countries. Science Media Centre. 27 Mar 2020. https://www.sciencemediacentre.org/expert-reaction-to-unpublished-preprint-paper-on-modelling-deaths-from-covid-19-after-implementing-social-distancing-measures-in-different-countries.
- ↵Yang G, Tan Z, Zhou L, et al. Angiotensin II receptor blockers and angiotensin-converting enzyme inhibitors usage is associated with improved inflammatory status and clinical outcomes in covid-19 patients with hypertension. medRxiv [preprint]. doi:10.1101/2020.03.31.20038935.Abstract/FREE Full TextGoogle Scholar
- ↵Expert reaction to preprint on COVID-19, high blood pressure, and high blood pressure drugs. Science Media Centre. 6 Apr 2020. https://www.sciencemediacentre.org/expert-reaction-to-preprint-on-covid-19-high-blood-pressure-and-high-blood-pressure-drugs.
- ↵Zhao J, Yang Y, Huang H. Relationship between the ABO blood group and the covid-19 susceptibility [preprint]. medRxiv. 27 Mar 2020. https://www.medrxiv.org/content/10.1101/2020.03.11.20031096v1.
- ↵Expert reaction to a preprint on blood type A and COVID-19 susceptibility. Science Media Centre. 18 Mar 2020. https://www.sciencemediacentre.org/expert-reaction-to-a-preprint-on-blood-type-a-and-covid-19-susceptibility.
- ↵
- Richardson S,
- Hirsch JS,
- Narasimhan M,
- et al.,
- and the Northwell COVID-19 Research Consortium
- ↵Langreth R. New study shows nearly 9 in 10 covid-19 patients on ventilators don’t make it. Bloomberg. 22 Apr 2020. https://www.bloomberg.com/news/articles/2020-04-22/almost-9-in-10-covid-19-patients-on-ventilators-died-in-study.
- ↵Clarification of mortality rate and data in abstract, results, and table 2. JAMA2020;24. doi:10.1001/jama.2020.7681. pmid:32330939CrossRefPubMedGoogle Scholar
- ↵Ferguson NM, Laydon D, Nedjati-Gilani G. Report 9: Impact of non-pharmaceutical interventions (NPIs) to reduce COVID-19 mortality and healthcare demand. Imperial College COVID-19 Response Team. https://www.imperial.ac.uk/media/imperial-college/medicine/sph/ide/gida-fellowships/Imperial-College-COVID19-NPI-modelling-16-03-2020.pdf.