.

Claims that a “Johns Hopkins study” showed lockdowns are ineffective at reducing COVID-19 mortality are based on a working paper with questionable methods

CLAIM
“Lockdowns only reduced COVID-19 mortality by .2%, study finds”
VERDICT more about the rating framework
DETAILS
Misrepresents source: The working paper’s analysis included some studies that actually concluded lockdowns to be beneficial in reducing COVID-19 mortality. But the paper’s authors instead represented these studies as showing that lockdowns were detrimental.
Misleading: Several reports about the working paper labeled it as a “Johns Hopkins study”. However, only one of the paper’s three authors is affiliated with Johns Hopkins University and the paper wasn’t endorsed by the university. Experts also pointed out certain issues in the paper’s methodology that call the reliability of its conclusions into question.
KEY TAKE AWAY
A lockdown is a non-pharmaceutical intervention that is typically defined as a measure that requires people to stay at home and avoid activity outside the home involving public contact. A lockdown can comprise different restrictions, ranging from stay-at-home orders to business closures. The type of restrictions implemented and severity of a lockdown can differ greatly between countries. Published scientific studies found that lockdowns are effective at reducing COVID-19 spread and mortality.

FULL CLAIM: “Lockdowns only reduced COVID-19 mortality by .2%, study finds”; “Lockdowns, school closures and limiting gatherings only reduced COVID mortality by 0.2% at 'enormous economic and social costs', study finds”; Lockdowns are “useless”

REVIEW


In early February 2022, multiple social media posts and articles published by various outlets, including Fox News (see here and here) and the Daily Mail, widely touted a “Johns Hopkins study” purportedly showing that lockdowns only reduced COVID-19 mortality by 0.2% and that lockdowns should be rejected as a pandemic measure. These articles and posts went highly viral on social media; the Fox News and Daily Mail articles received a combined engagement exceeding 130,000. Political commentator Tomi Lahren also posted a video citing the study, in which she called lockdowns “useless”, drawing more than 160,000 views on Facebook.

The alleged study refers to a working paper posted by three economists, Jonas Herby, Lars Jonung, and Steve Hanke, titled “A Literature Review and Meta-Analysis of the Effects of Lockdowns on COVID-19 Mortality”. Herby is a special advisor at Center for Political Studies in Copenhagen, Denmark, Jonung is professor emeritus in economics at Lund University, and Hanke is a professor of applied economics at the Johns Hopkins Institute for Applied Economics, Global Health, and the Study of Business Enterprise.

As the title of the paper explained, the authors undertook a literature review and meta-analysis. In simple terms, a literature review surveys the existing scientific work on a subject. It can be used to provide an overview of developments in a field and/or offer new insights based on previous studies, among other things. A meta-analysis combines data from multiple studies to answer a specific question, for example, whether sunscreen reduces the risk of skin cancer. One of the key advantages of a meta-analysis is that it provides greater statistical power, as it involves many more subjects in the analysis than a single study could provide.

However, several reports about the paper described it in inaccurate and misleading terms. More importantly, public health experts have drawn attention to issues in the authors’ methodology that call the validity of the findings into question, but aren’t acknowledged in certain reports.

The study is a working paper that isn’t peer-reviewed; authors of the paper are economists with no expertise in epidemiology

The paper was described by several outlets as a “Johns Hopkins study”. This description is problematic for several reasons. One reason is that only one of the three authors, Steve Hanke, is affiliated with Johns Hopkins University. The second is that the study wasn’t endorsed by the university. Finally, describing the paper as a “study” could mislead readers into believing that the paper is a peer-reviewed published study, which it is not. Even the authors themselves described it as a working paper, meaning that it is still a work in progress and hasn’t been peer-reviewed by other scientists.

Nevertheless, the label “Johns Hopkins study” stuck, likely because it burnished the paper’s credibility in the eyes of the public.

Secondly, many articles and social media posts promoting the study failed to inform readers that the authors are economists and lack expertise in epidemiology and infectious diseases, which is pertinent to this subject, given that lockdowns were used to combat the spread of an infectious disease.

The paper contained an unusual definition of a lockdown and methodological issues

Experts told the Science Media Center in the U.K. that the paper contained important methodological flaws.

One issue with the paper is that the meta-analysis it undertook “systematically excluded from consideration any study based on the science of disease transmission”, said Seth Flaxman, an associate professor at Oxford University who specializes in statistics.

“[T]he only studies looked at in the analysis are studies using the methods of economics. These do not include key facts about disease transmission such as: later lockdowns are less effective than earlier lockdowns, because many people are already infected; lockdowns do not immediately save lives, because there’s a lag from infection to death, so to see the effect of lockdowns on Covid deaths we need to wait about two or three weeks,” Flaxman explained.

Another point of contention was the paper’s definition of a lockdown. A lockdown is typically defined as a measure that requires people to stay at home and avoid activity outside the home involving public contact. However, the authors defined a lockdown as “the imposition of at least one compulsory, non-pharmaceutical intervention (NPI)”. Non-pharmaceutical interventions are measures apart from taking medicine and vaccination. This means that simply making isolation for infected people mandatory or imposing a mask mandate would count as a lockdown.

Neil Ferguson, an epidemiologist and professor at Imperial College London, pointed out that “by that definition, the UK has been in permanent lockdown since 16th of March 2021, and remains in lockdown – given it [remains] compulsory for people with diagnosed COVID-19 to self-isolate for at least 5 days.”

Samir Bhatt, a professor of statistics and public health at Imperial College London, likewise expressed similar concerns about the definition used in the paper, saying, “The most inconsistent aspect is the reinterpreting of what a lockdown is […] For a meta-analysis using a definition that is at odds with the dictionary definition (a state of isolation or restricted access instituted as a security measure) is strange.”

That being said, when it came to their analysis, the authors applied the usual definition of a lockdown, as Bhatt pointed out: “The authors then further confuse matters when in Table 7 they revert to the more common definition of lockdown.”

Epidemiologist Gideon Meyerowitz-Katz, who discussed the paper in a Twitter thread, also noted the confusion in the paper, saying, “The authors of this review define a ‘lockdown’ as LITERALLY ANY INTERVENTION”. But he also observed “this is certainly not how the analysis is actually done. In most cases, it seems the authors have taken estimates for stay-at-home orders as their practical definition of ‘lockdown’”.

But if we were to go with the authors’ initial definition of a lockdown, which is the application of any NPI, then “the reality is that virtually all research shows a (short-term) mortality benefit from at least some restrictions,” concluded Meyerowitz-Katz.

In fact, the working paper itself demonstrated this in their review:

In addition, Meyerowitz-Katz noted the paper’s anomalous results which were at odds with the studies that made up their meta-analysis. For example, one study[1] by Chisadza et al. included in the meta-analysis, which also contributed the greatest weight in the analysis, reported that lockdowns were beneficial with mitigating COVID-19-related deaths:

“This study investigates the effectiveness of current government policy responses to COVID-19-related deaths. We find that the overall government response index has a non-linear association with the number of deaths—driven by the containment and health interventions—for the aggregated sample of countries. The number of deaths increases with partially relaxed lockdown restrictions, but decreases with severe restrictions.”

Yet the working paper arrived at the opposite conclusion:

In fact, an author of the study by Chisadza et al. told science journalist Tom Whipple that the working paper’s authors had simply set out to prove a predetermined conclusion:

The same issue cropped up with another study[2], which found “strong evidence consistent with the idea that employee mask policies, mask mandates for the general population, restaurant and bar closures, gym closures, and high-risk business closures reduce future fatality growth”. But the study was somehow “included in this review as estimating a 13.1% INCREASE in fatalities”:

Finally, results from published studies contradict the findings of the paper, finding that lockdowns were beneficial in reducing the spread of COVID-19[3-7] and saving lives[2,8]. For example, Lurie et al. reported that “Among states without stay-at-home orders, the median increase in [epidemic] doubling time was 60% (95% CI, 9.2–223.3), compared with 269% (95% CI, 277.0–394.0) for states with stay-at-home orders”, suggesting that “stay-at-home orders combined with varied levels of implementing practices of testing, tracing, and isolation … as well as travel restrictions, likely played a key role in significantly reducing the epidemic growth rate”[3]. Sethman et al. estimated that across 11 countries in Europe, lockdowns helped prevent more than three million deaths[8].

If we consider the nature of infectious diseases, it is only to be expected that fewer cases and deaths follow a lockdown. Given that close contact with other people facilitates virus transmission, measures that limit such contact, like a stay-at-home order, would inevitably reduce infection and in turn deaths. An assertion to the contrary would require strong evidence, which the working paper doesn’t provide.

At least two of the three authors are known to have publicly expressed opposition to or skepticism of lockdown measures, suggesting that they already held certain preconceived notions about lockdowns and that their working paper was influenced by their preexisting beliefs. Readers should be made aware of this initial bias. Jonung co-authored an article titled “A Message from Sweden: Mandatory Lockdowns Do more Harm than Good” in Svenska Dagbladet, a Swedish newspaper. The first author Hanke, in particular, equated lockdown measures to fascism and falsely claimed that Vietnam didn’t report COVID-19 data (see screenshots below).

Screenshots of tweets by Hanke in June 2020 (left) and December 2021 (right) (archives: here and here).

Following his tweet about Vietnam, 285 “public health researchers and professionals and concerned citizens” penned an open letter to Johns Hopkins University demanding an apology from Hanke; Hanke later deleted the tweet.

In summary, there are several issues with the working paper that call into question the validity of its findings. Firstly, the paper’s definition of a lockdown (imposition of any NPI) differs from the dictionary definition of a lockdown. This distinction made in the working paper is lost in many articles and social media posts. Secondly, it’s unclear how the paper arrives at some of the figures it reported. As explained above, there was more than one instance where the paper provided an interpretation of a study included in their analysis that was at odds with what the study actually reported. However, certain outlets that reported on the working paper, such as Breitbart, the New York Post, and The Blaze, didn’t provide readers with context about the paper’s limitations and the issues raised about its methods, potentially leaving readers with a false impression about the reliability of the paper’s conclusions.

Overall, the claim that lockdowns are “useless” or ineffective at reducing COVID-19 mortality using this working paper as scientific justification is, in reality, unfounded, and contradicts the conclusions from published studies.

UPDATE (14 February 2022):

We added context regarding two of the working paper authors’ views on lockdown measures.

REFERENCES

   

Published on: 10 Feb 2022 | Editor:

Health Feedback is a non-partisan, non-profit organization dedicated to science education. Our reviews are crowdsourced directly from a community of scientists with relevant expertise. We strive to explain whether and why information is or is not consistent with the science and to help readers know which news to trust.
Please get in touch if you have any comment or think there is an important claim or article that would need to be reviewed.

ifcn-fact-checkers-code-of-principles-signatory