FULL CLAIM: By administering hydroxychloroquine combined with azithromycin, they were able to observe an improvement in all cases, except in one patient
SUMMARY
On April 6 2020, a website named Tech Startups published a piece that reported partial results of a study testing a combination of hydroxychloroquine (HCQ) and the antibiotic azithromycin as a treatment against COVID-19. The study by Didier Raoult, an infectious disease specialist at the IHU-Méditerranée Infection in Marseille, France, and colleagues was first released in late March 2020 and published in the journal Travel Medicine and Infectious Disease on 11 April 2020. In part, it reports the clinical outcomes, such as length of hospital stay and contagiousness, of 80 “relatively mildly infected” COVID-19 inpatients who received the treatment for at least three days. The Tech Startups article stated that “by administering hydroxychloroquine combined with azithromycin, they were able to observe an improvement in all cases, except in one patient who arrived with an advanced form”.
The study found that 77 of the 80 patients treated showed clinical improvement by the end of the study. For one patient, the treatment had to be stopped on day 4 due to potential risk of interaction with another drug (the trial was planned to last 10 days). A second patient died and a third still remained in intensive care at the end of the study, so that individual’s outcome was unknown at the time of the study’s publication.
However, the high percentage of improvement observed in the patients is undermined by the study’s numerous design flaws, as identified by members of the biomedical community#. Furthermore, the study does not establish a causal relationship between the HCQ/azithromycin treatment and the improved clinical outcomes. And finally, the U.S. Food and Drug Administration also recently cautioned “against use of hydroxychloroquine or chloroquine for COVID-19 outside of the hospital setting or a clinical trial due to risk of heart rhythm problems.”
In order to assess the effectiveness of a given treatment, a study must include a control group that receives the standard of care instead of the experimental treatment. Ideally, this control group must contain individuals representing the same demographics and medical backgrounds as the experimental treatment group. This ensures that confounding factors will not interfere with the outcome of the clinical trial. For example, COVID-19 is known to cause more serious illness in the elderly population compared to younger patients, making age a confounding factor. Having a group with more elderly patients than the other group would make it difficult to discern whether treatment outcome was influenced by the effects of an experimental drug or by patient age.
In the absence of a control group, it is therefore impossible to determine whether the observed improvement was due to HCQ/azithromycin or was simply the natural course of the disease. Derek Lowe, a chemist with a background in drug discovery, writes in Science Translational Medicine: “Without matched controls, and without being able to look at individual patient data, we just don’t know how good this treatment was or frankly if it was any good at all.”
Natalie Dean, a biostatistician who specializes in clinical trial design and is a professor at the University of Florida, agrees: “Without a randomized control group, it is very hard to determine what would have happened to patients in the absence of treatment. Many would have recovered on their own.”
An alternative, albeit suboptimal, approach to including a control group, would be to compare the clinical outcomes of the patients treated with HCQ/azithromycin with the outcomes of patients not treated with the drug combination in hospitals elsewhere. However, this approach is not entirely effective for several reasons. First, some key metrics required for such a comparison are difficult if not impossible to calculate between hospitals, countries, or even the time of year, because they are heavily influenced by local policies and resource availability.
Key metrics include the case fatality ratio (CFR) and the proportion of patients requiring oxygen assistance or transfer to ICU. For instance, the CFR is calculated based on the number of known cases. Given that cases are often asymptomatic, many remain undetected in the absence of systematic testing of the population. As a result, CFRs vary greatly by hospital and region[1]. The literature reports, for instance, CFRs for COVID-19 that range between 0.4% and 1.38%[1,2].
In contrast, epidemiologists were able to obtain a robust CFR estimate of 0.99% on the Diamond Princess cruise ship[1] due to the small population size and quarantine status of the passengers, which enabled scientists to closely track infected, uninfected, and asymptomatic passengers alike. With one death out of 80 patients, Raoult’s study found a rather similar CFR of 1.3%. However, the authors reported in a follow-up paper, described below, that a second patient also died after the end of the study, bringing the CFR to 2.5%. Therefore, the CFR among the cohort in this study is not significantly different from what has been observed so far for COVID-19. It is thus not possible to conclude from the study that the treatment would have impacted fatality rate. In addition, we know that the disease affects people differently depending on their age or preexisting conditions; the study does not provide enough information about the demographics of the individuals included or excluded, nor about the pool of individuals from which they were selected.
Other flaws in the study’s design further complicate the ability to draw conclusions from the results. For one, the study does not present data at the single-patient level, which makes it impossible to track progression of the disease for each patient. Specifically, the study does not present the viral load of each patient at the onset of the treatment. Viral loads are determined using a technique called reverse transcription polymerase chain reaction (RT-PCR), which yields a so-called Ct value. The lower the Ct value, the higher the viral load (inverse correlation). To compare the outcomes of patients receiving HCQ/azithromycin treatment with the outcomes of patients in different hospitals who did not receive the treatment, one would have to compare patients with similar initial viral loads. Yet the study from Raoult’s team provided only aggregated statistics of the Ct value (median, minimum, maximum). It is therefore impossible to track changes in the viral load for individual patients.
In further support of the claim that HCQ/azithromycin is an effective treatment against COVID-19, the Tech Startups article also mentions a previous study by the same research group, stating that “French researchers conducted a study which showed that 100% of coronavirus patients that received a combination of hydroxychloroquine and azithromycin were virologically cured within six days of treatment”. However, the article is misleading because it fails to mention that this first study received much criticism@ from members of the scientific community for design flaws including small cohort size and lack of a control group.
In addition, a follow-up study with 1,061 patients conducted by Raoult’s group and posted on his institution’s website on 20 April 2020 reported a “good clinical outcome and virological cure” in 91.7% of the cohort. This third study by Raoult and colleagues contains several similar study design limitations, including no control group, no alternative attempt to compare the outcome of patients in the study to patients in other hospitals who did not receive the treatment, and the inclusion of a large number of patients with low National Early Warning Scores (NEWS), which are standardized scores established in the U.K. that indicate the severity of a patient’s illness$.
In summary, the Tech Startups article presents the results of three fundamentally flawed clinical studies as evidence that the HCQ/azithromycin treatment was effective in combating COVID-19. Specifically, the article omits criticism from credible members of the scientific community which illustrates the impossibility of drawing conclusions from the study on the efficacy of the treatment. As a result, the article’s claim that the HCQ/azithromycin treatment was responsible for the observed clinical improvements in 77 of 80 patients is misleading.
The use of HCQ alone and in combination with other drugs to treat COVID-19 is currently the focus of intense debate, both scientifically and politically. But many clinical trials are underway which may shed more light on the potential effects of such drugs. So to avoid misleading the public while the scientific community continues to gather evidence on this topic, articles that report the findings of preliminary studies must appropriately point out the limitations of those studies and also calibrate the public’s confidence in the findings by placing it in context with the broader body of existing peer-reviewed research.
SCIENTISTS’ FEEDBACK
Natalie Dean, Assistant Professor, University of Florida:
I haven’t looked into the details of the paper as close as some have, but in general, without a randomized control group, it is very hard to determine what would have happened to patients in the absence of treatment. Many would have recovered on their own. And it is hard to determine an expected rate of recovery from a “comparable” group because recovery varies by so many factors (age, presence of comorbidities), and others that we may not yet fully understand (sex? infectious dose?). There were also some exclusions from the analysis set that weren’t well explained, and this is part of the reason why many have called for the paper to be retracted.
NOTES
#: Aside from the small size of the cohort, the main criticism has been the lack of a control group in the study. Concerns have been raised about the way the viral load was monitored, as well as the cohort composition (large number of people with good prognosis to begin with). See for example here and here, and on Twitter here, here, and here.
@: Critics focused mostly on the small size of the cohorts, the fact that test patients and control patients were treated in different hospitals, inconsistencies in the RT-PCR measurements of viral load, and statistical flaws that preclude any conclusions. See for examples of these critiques here and here. Scientists also took to Twitter to expose the limits and flaws of the study, see here, here, here, and here for examples.
&: Raoult’s team defined a patient with a low risk of clinical deterioration as having a NEWS score below 4, which accounted for 95% of the patients in the study.
REFERENCES
- 1 – Rajgor et al. (2020) The many estimates of the COVID-19 case fatality rate. The Lancet Infectious Disease.
- 2 – Verity et al. (2020) Estimates of the severity of coronavirus disease 2019: a model-based analysis. The Lancet Infectious Disease.
