Misleading: The Florida Surgeon-General’s tweet and the press release by the Florida Department of Health left out the fact that the analysis actually detected lower all-cause mortality in vaccinated people across all age groups, and that the increase in cardiac-related deaths observed in the general population actually disappeared once those aged 60 and above were excluded from the analysis.
FULL CLAIM: Florida Department of Health analysis found “an increased risk of cardiac-related death among men 18-39” who received COVID mRNA vaccines; “the benefit of vaccination is likely outweighed by this abnormally high risk of cardiac- related death among men in this age group”; “Covid mRNA vaccine found to cause 84% increase in DEATH for men ages 18-39”
REVIEW
On 7 October 2022, the state of Florida issued new COVID-19 vaccine guidance recommending against men aged 18 to 39 years old getting the COVID-19 mRNA vaccines. According to the guidance, this decision was based on a new unpublished study by the Florida Department of Health, which Florida Surgeon-General Joseph Ladapo claimed on Twitter had found “an increased risk of cardiac-related deaths” in men within that age group associated with mRNA vaccines.
The analysis, whose authors are unnamed, examined the death records of Florida residents aged 18 years or older who died within 25 weeks of COVID-19 vaccination since the vaccines were first rolled out in December 2020. Those who’d had documented COVID-19, as well as those who died of COVID-19, were excluded from the analysis.
The authors calculated natural all-cause deaths (excluding COVID-19 deaths) and cardiac-related deaths, and reported that “In the 28 days following vaccination, a statistically significant increase in cardiac-related deaths was detected for the entire study population”, and that males aged 18 to 39 exhibited the highest risk.
Several websites, including The Daily Wire, The Post Millennial, and the Western Journal, publicized the findings of the analysis as well as the tweet by Ladapo. On 11 October 2022, Ladapo also appeared on Fox News, repeating the same claim he’d made in his tweet.
It’s not the first time that the government of Florida has presented COVID-19 data in a misleading manner. In October 2022, governor Ron DeSantis claimed that “Florida is one of the big COVID success stories” as it had the one of the lowest daily case and death rates in the U.S. FactCheck.org reported that while Florida’s daily COVID-19 case rate was low at that time, “Florida’s total case rate [was] still higher than the national rate and among the highest in the U.S.”
Scientists have criticized the Florida study’s methods and found the way it was presented as evidence of vaccine risk to be misleading. This review will explain the key limitations of the analysis that make it inadequate for supporting the claim.
The analysis violated a baseline assumption essential to the study design used, leading to bias
The study was designed as a self-controlled case series (SCCS), which was originally developed to study vaccine safety. In such a study, individuals act as their own controls, which is helpful for researchers as it automatically accounts for certain confounding factors, like age and a person’s state of health[1].
To determine whether a vaccine is associated with a higher risk of an adverse event, researchers compare the incidence of the adverse event between a defined period following vaccination (risk period) with that of the period in which the vaccine isn’t expected to influence the risk of the adverse event (baseline period). Incidence is the number of events divided by a specified period of time.
In the case of this study, individuals’ records were examined for 25 weeks following vaccination; the period at which individuals were considered to be at risk was 28 days post-vaccination.
Figure 1. A schematic showing the study design. The red dot marks the time of the individual’s death. The incidence of deaths during the risk period would be compared with the incidence during the baseline period to see if there was a notable difference between the two. Note that this schematic is for illustrative purposes only; the analysis included far more than six individuals and the deaths depicted in the schematic aren’t intended to reflect actual deaths analyzed.
As with any study design, there are certain baseline assumptions that need to be fulfilled for researchers to avoid bias and obtain valid results. One of these is that the occurrence of the adverse event in question shouldn’t affect later possible events.
However, the adverse event of interest in the analysis is death, which violates this assumption. As Deepti Gurdasani, an epidemiologist and senior lecturer at Queen Mary University of London, pointed out, someone who died during the risk period cannot die in the baseline period. This means that they are no longer at risk of death during the baseline period.
In spite of this, the analysis continued to include people who had died in the risk period when calculating the incidence of death during the baseline period (“Participants were not censored upon death, rather, they were followed for the entire 25-week follow-up period”). This led the authors to underestimate the baseline incidence of death. A detailed explanation of why this inclusion becomes a problem was illustrated by Gurdasani in this tweet.
Consequently, when the risk period incidence was compared with that of the baseline period, the incidence during the risk period appeared larger, making it more likely for the authors to find that the incidence of cardiac-related death is much higher following vaccination.
The claim that males aged 18 to 39 have a higher risk of cardiac-related deaths is based on an analysis of a small number of deaths
With regards to the increase in cardiac-related deaths in males aged 18 to 39, Kristen Panthagani, an emergency medicine physician and scientist at Yale University, highlighted that the numbers analyzed in this subgroup were very small. In a tweet, she explained that “The results for males age 18-39 who received an mRNA vaccine included a grand total of 20 deaths during the risk period. If even just a few of these have another cause (which is highly likely), they lose statistical significance”.
Speaking to the Washington Post, Robert Wachter, a patient safety expert and professor at the University of California San Francisco, said the same: “We’re talking about a very small number of deaths. An extra death or two would potentially change these results”.
In other words, the deaths in the subgroup of 18 to 39-year-old men (52 deaths in the risk period compared to 20 deaths in the baseline period) are so small that conclusions made using this comparison could potentially change, if the numbers for either group are incorrect by even just one or two deaths.
Cardiac diagnoses weren’t verified with medical records, making their alleged relationship to COVID-19 vaccines uncertain
This leads us to the next question: did the study authors check whether the cardiac-related deaths they included in their analysis could have had some other cause apart from the vaccine? This would have gone some way towards mitigating the problem discussed in the previous section.
Panthagani pointed out that the authors identified cardiac deaths through hospital billing codes, but didn’t cross-check this information with medical records to see if the patient had another obvious cause for cardiac-related death, like bacterial pericarditis (an infection of the membrane lining the heart). This would have helped the authors know whether they could rule out the COVID-19 vaccine as a cause.
And the authors themselves acknowledged this as a limitation in their Discussion:
“This study cannot determine the causative nature of a participant’s death. We used death certificate data and not medical records. COVID testing status was unknown for those who did not die of/with COVID. Cardiac-related deaths were ascertained if an ACME code of I3-I52 were on their death certificate, thus, the underlying cause of death may not be cardiac-related.” [emphasis added]
Given this limitation, we don’t know whether the analysis incorrectly included cardiac-related deaths that may well have had other obvious explanations, and how much this influenced their analysis.
The analysis found lower all-cause mortality in vaccinated people, contradicting the claim that the COVID-19 mRNA vaccines’ risk outweighs the benefit
Epidemiologist Gideon Meyerowitz-Katz pointed out that the analysis actually found lower all-cause mortality or no difference in vaccinated people across all age groups. He questioned why Ladapo’s tweet never mentioned this observation.
6/n This may surprise you, because the way the study and press release have been written is…very deceptive. But all three of those statements are true
First point – the relative incidence of all-cause mortality was lower (sometimes significantly) after COVID-19 immunizations pic.twitter.com/nigt5GtlTN
— Health Nerd (@GidMK) October 9, 2022
Furthermore, Gurdasani highlighted that the study’s Discussion indicated that there were likely confounding effects from the people ages 60 and above. In addition, the authors stated that the majority of deaths included in the analysis were in this age group, likely because of their advanced age.
Indeed, the authors reported that once this group was removed from the analysis, they no longer saw an increase in cardiac-related deaths following vaccination:
“Removing those aged 60 years or older yielded non-significant results for cardiac-related deaths following vaccination (RI = 1.15, 95% CI = 0.99 – 1.34), mRNA vaccination (RI = 1.17, 95% CI = 1.00 – 1.37), and males with mRNA vaccination (RI = 1.09, 95% CI = 0.89 – 1.34).” [emphasis added]
However, nowhere is this indicated in the guidance or the Surgeon-General’s tweet. In short, the claim that COVID-19 mRNA vaccines increase cardiac-related deaths in general appears to be based on a statistical artifact, originating from the inclusion of the elderly population in the analysis, and not a true effect of the vaccine.
The analysis didn’t measure benefits from COVID-19 vaccination and therefore cannot provide a risk-benefit assessment
The Florida guidance claimed that the analysis showed “the benefit of vaccination is likely outweighed by this abnormally high risk of cardiac- related death among men in this age group”. However, the analysis excluded documented COVID-19 cases and COVID-19 deaths. There was no attempt to calculate benefits, such as hospitalizations prevented or lives saved. Therefore, the analysis is incapable of providing evidence for the claim stated in the guidance.
What’s often lost in claims focused on the harms due to heart inflammation post-vaccination is the fact that COVID-19 itself is more likely to lead to cardiac complications than COVID-19 vaccines, as studies from the U.S. Centers for Disease Control and Prevention and Oxford University showed[2,3].
The American Heart Association considers the benefits of the COVID-19 vaccines to outweigh their risks. Likewise, an expert consensus by the American College of Cardiology found that “a very favorable benefit-to-risk ratio exists with the COVID-19 vaccine for all age and sex groups evaluated thus far”[4].
Figure 2. There is a favorable benefit-to-risk ratio for COVID-19 mRNA vaccine recipients at highest risk for postvaccination myocarditis[5].
Conclusion
Overall, the Florida study provided inadequate evidence for the claim that vaccinated people, particularly men between 18 to 39 years old, are more likely to die from cardiac-related reasons. The number of deaths involved in the analysis of men in this age group was small, making it difficult to determine if the authors’ observation was simply a statistical fluke.
The analysis was also unable to demonstrate that the COVID-19 mRNA vaccines caused cardiac-related deaths, as the authors themselves conceded. It performed no risk-benefit analysis, therefore it didn’t show that the vaccines are riskier than they are beneficial, as some claimed.
Furthermore, the authors acknowledged that their observation of increased cardiac-related deaths following vaccination was no longer true after the elderly were excluded from their analysis, raising questions about the validity of the claim that the COVID-19 vaccines increase the risk of cardiac-related deaths in general.
Ladapo later published a Twitter thread “to respond to the more substantive critiques” about the analysis. However, none of his responses addressed the key flaws in the analysis that scientists identified and that this review discusses.
More often than not, viral social media posts about this analysis left out the fact that the analysis showed lower all-cause mortality in vaccinated people, meaning that on the whole, vaccinated people are actually less likely to die. Larger studies of better quality came to the opposite conclusion of this analysis: that the benefits of the COVID-19 mRNA vaccines outweigh their risks, even in those at the highest risk of post-vaccine myocarditis.
REFERENCES
- 1 – Petersen et al. (2016) Self controlled case series methods: an alternative to standard epidemiological study designs. BMJ.
- 2 – Block et al. (2022) Cardiac Complications After SARS-CoV-2 Infection and mRNA COVID-19 Vaccination — PCORnet, United States, January 2021–January 2022. Mortality and Morbidity Weekly Report.
- 3 – Patone et al. (2022) Risk of Myocarditis After Sequential Doses of COVID-19 Vaccine and SARS-CoV-2 Infection by Age and Sex. Circulation.
- 4 – Writing Committee. (2022) 2022 ACC Expert Consensus Decision Pathway on Cardiovascular Sequelae of COVID-19 in Adults: Myocarditis and Other Myocardial Involvement, Post-Acute Sequelae of SARS-CoV-2 Infection, and Return to Play: A Report of the American College of Cardiology Solution Set Oversight Committee. Journal of the American College of Cardiology.
- 5 – Bozkurt et al. (2022) Myocarditis With COVID-19 mRNA Vaccines. Circulation.
