2
$\begingroup$

I think I found a major (conclusion-flipping) statistical error in a paper in an AMA journal. Did I?

If I messed up, I'd like to know how; I hope someone can point me in the direction of my errors. If I messed up, I must have made at least two major errors, as I came to the same conclusion in two independent ways. I communicated with the journal editor and corresponding author.

Here, you can find the paper and the correspondence. I reproduce it below.

To try to make this question fully self-contained, I'll summarize the issue.

The authors calculate the background rate of sudden sensorineural hearing loss (SSNHL) per year and compare it to the rate of SSNHL over a three-week post-intervention period, and graph "Estimated incidence of SSNHL, per 100 000 per y". Their conclusion is that the data indicates the intervention does not increase the incidence of SSNHL; a substantially and significant reduction is indicated.
They state that "We then estimated the incidence of SSNHL that occurred after vaccination on an annualized basis." But this cannot be what they calculated. It is incompatible with what they report is the data their research yielded.

  1. It’s an error to limit the possible adverse side effect window to 3 weeks post-vaccination (excluding adverse events outside that window) but then spread the remaining adverse events over a year to calculate risk on an annualized basis.". It’s unjustifiable. A reasonable start for comparison would be to compare risk over the 3 weeks to the annual (52-week) risk, scaled to 3 week period. So the correct finding, based on their research, appears to be no risk difference over a 3 week period of SSNHL between groups, (0.6-4.4 vs 0.3-4.1, n.s.).

  2. Their conclusion implies that the authors have discovered that the intervention reduced SSNHL by about 94%. Which would be a groundbreaking discovery if confirmed, and a there’s no plausible mechanism presented for such a miraculous treatment effect, more evidence of grave error. It does not pass this basic plausibility test.

  3. As I finished writing this up, I found further concerns, which I'll put in an answer. I put in chat: https://chat.stackexchange.com/rooms/18/ten-fold because preliminary.

[end summary]

Again, here, you can find the paper and the correspondence. I reproduce it immediately below.

I wrote: (slightly modified)

Sirs:

I write in respect to *.

I may have found an error in this paper. I'm thinking: It’s an error to limit the possible adverse side effect window to 3 weeks post-vaccination (excluding adverse events outside that window) but then spread the remaining adverse events over a year to calculate risk. It’s unjustifiable. A reasonable start for comparison would be to compare risk over the 3 weeks to the annual (52-week) risk, scaled to 3 week period. So the correct finding appears to be that risk in a 3 week period of SSNHL, whether vaccinated or unvaccinated, is the same (0.6-4.4 vs 0.3-4.1, n.s.). A closer look at adverse events within shorter periods after vaccination would be an appropriate topic for further research. Another way to see this error is to consider whether the original results pass a basic plausibility test. They do not. If the results shown in the figure accurately reflected Incidence Range / 100k of SSNHL between the vaccinated and unvaccinated, then it would suggest that the authors had discovered that vaccination reduced SSNHL by about 94%. Which would be a groundbreaking discovery, and a there’s no plausible mechanism presented for such a miraculous treatment effect, this is more evidence of grave error.

*Formeister EJ, et. al. JAMA Otolaryngol Head Neck Surg. 2021;147(7):674–676. doi:10.1001/jamaoto.2021.0869

-Matthew

Assuming I have, this won't my first time spotting a major error in a peer-reviewed publication. (I think that was in article High-fructose corn syrup causes characteristics of obesity in rats: Increase body weight, body fat and triglyceride levels (2010) in 2010. This HFCS, Bocarsly, Princeton paper was wildly popular in the lay press.)

Yet I received this non-response response (emphasis mine):

Matthew,

Thank you very much for your recent communication about the paper w published in JAMA Otolaryngology ("Preliminary Analysis of Association Between COVID-19 Vaccination and Sudden Hearing Loss Using VAERS"). As a peer reviewed publication this manuscript was vetted in a process that includes assessment and validation of hypotheses, methodologies, and conclusions. Readers and scientists can have faith in the integrity of these robust processes. We look forward to seeing this important field expand and would encourage all interested scientists to consider peer reviewed publication of their work in the field. We would encourage a thoughtful re-read of the manuscript to understand the methodology, and additional reading on the topics of idiopathic sudden sensorineural hearing loss and principles of epidemiology, for your understanding. Respectfully, Dr. Eric Formeister, MD, MS on behalf of the authors.

-- If I messed up, I'd like to know ; I hope someone can point me in the direction of my error(s).

$\endgroup$
6
  • 1
    $\begingroup$ It is not necessary to collect data over a whole year to calculate an annualised rate. $\endgroup$ Aug 18 '21 at 20:47
  • 1
    $\begingroup$ Of possible interest: fharrell.com/post/errmed $\endgroup$
    – Dave
    Aug 18 '21 at 21:03
  • $\begingroup$ Michael, of course not. Not saying that. One could *52/3, roughly. (Similar to what I said, "A reasonable start for comparison would be to compare risk over the 3 weeks to the annual (52-week) risk, scaled to 3 week period." *3/52, roughly). But one has to actually do it. $\endgroup$ Aug 18 '21 at 21:05
  • $\begingroup$ Added some preliminary further thoughts to chat: chat.stackexchange.com/rooms/18/ten-fold $\endgroup$ Aug 18 '21 at 22:37
  • $\begingroup$ Usually peer-reviewed research is disproved by other peer-reviewed research. Your initial message seems far too short and cursory to communicate what you are trying to communicate in a way that would be generally accepted. They won't take the time to investigate your claim if you only send a one paragraph email. I don't know if this is the case or not, but it also seems to be generally good practice to know the major findings of everything an article is citing before challenging it. $\endgroup$ Aug 22 '21 at 1:23
12
+50
$\begingroup$

It's exciting to think that one has seen a fatal flaw in some published work. But sometimes that vision of a "fatal flaw" is a mirage, arising from a misunderstanding of the question at hand, the nature of the study, or its interpretation.

Sudden sensorineural hearing loss

Sudden sensorineural hearing loss (SSNHL) is an "unexplained, rapid loss of hearing—usually in one ear—either at once or over several days." Should this happen to you, get medical treatment immediately as "eighty-five percent of those who receive treatment from an otolaryngologist (sometimes called an ENT surgeon) will recover some of their hearing." Treatment typically involves steroids, best given as soon as possible after the hearing loss is noted.

About 90% of cases are idiopathic; that is, the hearing loss has no identifiable cause. A viral infection can sometimes be identified as a cause. The SARS-Cov-2 virus that causes COVID-19 might be among such viruses.

Alexander and Harris estimated the annual incidence of SSNHL in the US by examining medical and pharmaceutical claims information from more than 60 million patients:

During 2006 and 2007, the annual incidence of SSNHL was 27 per 100,000 in the United States. The incidence increased with increasing age, ranging from 11 per 100,000 for patients younger than 18 years to 77 per 100,000 for patients 65 years and older.

That range of 11 to 77 cases per 100,000 per year is shown in the Figure of the paper you cite by Formeister et al.

COVID-19 vaccination and SSNHL

As Formeister et al reported:

Anecdotal reports are rapidly emerging from the otolaryngology community of SSNHL occurring after inoculation by SARS-CoV-2 vaccines that are currently in use in the US under US Food and Drug Administration Emergency Use Authorizations.

The authors noted patients coming to their own practice to report SSNHL starting within 24 hours of COVID-19 vaccination. Their practice, a major referral center, tends to be over-represented with difficult problems like SSNHL. Together with the small numbers of SSNHL cases, preparing single-institution case reports on this matter would not have been very useful.

The goals

The critical clinical issue facing the authors was whether nationwide SSNHL incidence following COVID-19 vaccination might be greater than expected from the historical incidence of SSNHL, representing an unexpected side effect of vaccination. Their goals were

to (1) estimate the national incidence of SSNHL after COVID-19 vaccination using data from the Vaccine Adverse Events Reporting System (VAERS) maintained by the US Centers for Disease Control and Prevention (CDC) and (2) compare this with the expected incidence of SSNHL in the wider population.

VAERS is the repository for patients and clinicians to report suspected side effects from vaccination.

The findings

The authors found 127 unique reports of potential SSNHL reported to VAERS between December 14, 2020 and March 2, 2021, the first 3 months of widespread COVID-19 vaccination in the US. During that period, 86,553,330 vaccine doses were administered. With the Pfizer and Moderna vaccines then available, 2 doses were possible per individual. The total number of doses sets an upper limit on the number of people at risk for vaccine-related SSNHL; the lower limit (assuming 2 doses for all individuals) is half that: 43,276,665.

The authors reported post-vaccine SSNHL incidence estimates between 0.3 (minimum) and 4.1 (maximum) cases per 100,000 per year, respectively. Even the maximum incidence estimate was well below the range of normal SSNHL incidence.

Authors' conclusions

the incidence of SSNHL occurring after COVID-19 vaccination does not exceed that of the general population, and may be lower.

That provided one less thing to worry about in the midst of a pandemic that was taxing the medical profession nationwide and worldwide.

The authors certainly did not claim that vaccination prevented SSNHL. There is no reason to put that claim into their mouths.

Sources of the annualized estimates

In their brief report (~700 words, about half the length of this answer), the authors were unfortunately vague about some details of the "sensitivity analysis" that led to those estimates. In particular, it's not clear (a) what's meant by "100% VAERS underreporting bias" in deriving the maximum incidence estimate or (b) the observation period used to calculate annualized incidence.

In a situation like this, it can help to try to reverse-engineer the reported result to see what specific assumptions might have been made.

Their minimum-incidence estimate of 0.3 per 100,000 per year, applied to the corresponding assumed population of 86,553,330, works out to 260 estimated cases per year in that population. I suspect that the estimate of 0.3 was rounded up from 0.277, which would work out to 240 cases per year in that population. That estimate was based on the 40 reports in VAERS that were most likely to have been associated with vaccination, assuming all cases were reported to VAERS (no reporting bias). Thus this annualized estimate is 6 times the corresponding numbers reported during the study period.

Their maximum-incidence estimate of 4.1 per 100,000 per year, applied to the corresponding assumed population of 43,276,665, works out to 1774 estimated cases per year in that population. That estimate was based on all 147 unique cases of hearing loss reported to VAERS.* This annualized estimate is about 12 times the corresponding numbers reported during the study period.

(a) Put those extreme estimates together in terms of the ratio of estimated cases per year to corresponding observed numbers: 6/1 for the minimum-incidence estimate, 12/1 for the maximum-incidence estimate. Then the authors evidently meant by "100% VAERS underreporting bias," in the maximum-incidence estimate, a situation with only 1/2 of SSNHL cases reported to VAERS. Then the VAERS-reported value of 147 used for the maximum-incidence estimate would have represented 294 "actual" cases--again, a ratio of annual incidence to "actual" numbers of 6/1.

(b) So how do you get from an observed (or bias-corrected) number of cases to 6 times that many per year? To get the annualized rate you need to know the observation period. Reports would only be made to VAERS after a vaccination, so the numbers under observation for vaccine-associated SSNHL increased over the 3 months of data examined by the authors. The observation period is longer for those vaccinated in December 2020 than for those vaccinated in February 2021. It seems that the authors assumed 2 months for an average observation time, for annual numbers 6 times the observed (or bias-corrected) numbers.

Sensitivity to assumptions

There clearly are several assumptions at work here. For example, I might have chosen 1.5 months for the average observation period (one half of the 3 months of data collection, assuming that individuals were added linearly over time as they got their first vaccine doses). That would give a maximum-incidence estimate of 5.5 per 100,000 per year. Or if you prefer 3 weeks for an average observation period (which seems low), then 11 per 100,000 per year following vaccination as maximum incidence.

Perhaps a more important assumption is that at least half of SSNHL cases in vaccinated individuals would be reported to VAERS as a reaction to vaccination. What if that's an overestimate? How much does that matter?

For that, let's go back to the estimates of SSNHL incidence by age from Alexander and Harris. The population getting vaccinated during the time period in question was primarily the elderly (with already high incidence of SSNHL) and healthcare workers. Let's use the incidence for 55-64 year-olds in that study as a guide to what might be expected of that population pre-vaccination: 47 per 100,000 per year (their Fig. 1).

That's 10 times the maximum-incidence post-vaccination estimate by Formeister et al. For COVID-19 vaccination to represent a higher than normal risk of SSNHL (the point of this study), they would need to have made an order-of-magnitude set of errors in their assumptions.

Summary

A question in early 2021 was whether COVID-19 vaccination was going to lead to a major increase in SSNHL incidence. This preliminary report was what one used to call a "back-of-the-envelope" calculation, a rough estimate to get a sense of the magnitude of a problem. It might be best to think of this as a first pass at a non-inferiority study, showing that SSNHL is certainly no more prevalent following vaccination than it is in the general population. That's the point of the paper.


*The authors did not "limit the possible adverse side effect window to 3 weeks post-vaccination," as you claim. They used that time window, among other data, as a guide to selecting the cases with the greatest likelihood of SSNHL causally related to vaccination. Those 40 cases are a basis only for the "minimum-incidence" estimate, which is of only secondary importance in this type of study. The clinically critical "maximum-incidence" estimate, needed to see if vaccination might be accounting for the anecdotal reports of SSNHL following vaccination, took all 147 reports of hearing loss during the reporting period and doubled that number to account for reporting bias.

$\endgroup$
8
  • $\begingroup$ Thank you for taking the time to provide such a detailed response! You've provided a great amount of detail on their paper. I have read halfway through your answer (up to the "Authors' conclusions" section) and have yet to see anything that addresses my question. You do re-explain their work well, but I understood it as well based on their explanation. It seems you missed that I saw the "11 per 100,000 for patients younger than 18 years to 77 per 100,000". The 0.6-4.4 range I referred to is as I described it - it's the 11-77 range scaled by 3 weeks / 52 weeks. ... STILL READING! $\endgroup$ Aug 28 '21 at 9:30
  • 1
    $\begingroup$ @MatthewElvey-RNSTTIVERMECTIN answers need to be intelligible to future visitors to the site, hence the detailed description for those who can't get access to it. The section on "Sources of the annualized estimates" answers the main part of your question. Low limit is based on the most-certain 40 cases, assuming all cases were reported and with a 2-month average observation window. Upper limit uses all 147 unique SSNHL reports, assuming only 1/2 of cases reported and the same 2-month average observation window. Sensitivity to those assumptions is analyzed thereafter. $\endgroup$
    – EdM
    Aug 28 '21 at 17:05
  • $\begingroup$ @MatthewElvey-RNSTTIVERMECTIN also assumed for the lower limit: numbers at risk equal the number of vaccine doses administered. For the upper limit: numbers at risk equal one-half of the number of vaccine doses administered, with each individual getting the 2 recommended doses. Note that the clinical interest here is in the upper-limit estimate: whether vaccination might be leading to increased SSNHL incidence. The authors' assumptions would have had to be off by an order of magnitude for that to be the case. $\endgroup$
    – EdM
    Aug 28 '21 at 17:13
  • $\begingroup$ @EdM- Did you read the chat I referred to? You didn't mention the huge discrepancy between what they and I found in VAERS. (Also, you said, "In a situation like this, it can help to try to reverse-engineer the reported result to see what specific assumptions might have been made." Did you notice that's exactly what I tried to do, as shown in the chat? $\endgroup$ Aug 29 '21 at 19:47
  • 1
    $\begingroup$ @MatthewElvey-RNSTTIVERMECTIN I don't know enough about VAERS to have an informed opinion about the apparent data discrepancy, which is outside statistics per se anyway. The authors did do some clinically informed data cleaning to remove duplicate reports, etc. I suspect that they were trying to set as wide limits as possible on both ends of the scale. Otolaryngologists reading this report would not have cared about the lower limit, as the critical question was whether vaccination was increasing SSNHL--which it wasn't, unless the authors were off by an order of magnitude. $\endgroup$
    – EdM
    Aug 30 '21 at 13:13
9
$\begingroup$

I write in respect to *.

This study should be withdrawn.

I'm not going to wade through the material here to form a view on whether you are right or wrong in your identification of an error. However, I will say that I don't think it's a good idea to start by requesting that the paper be withdrawn. The first appropriate action here is to write to the authors, explain your concern with the paper, and see if they agree that there is a mistake in the paper. They might agree with you (and then take appropriate action themselves, such as post errata or a withdrawal) or they might disagree and give you a counter-argument to your position, or they might just ignore you. Whatever happens, you will then be in a position where you can decide whether you want to proceed further, and you will have made an initial attempt to clarify the matter and resolve it with the authors before proceeding to take action with the journal.

If you start out guns-blazing requesting withdrawal, and it turns out you are wrong, you are going to look like a bit of a goose. Even if you are right, using this as your opening communique is not a very collegial approach. It is best to assume that academics are generally competent people who act in good faith and are concerned about the correctness of their work --- in cases where you think a paper might need to be withdrawn, it is usual to start by raising concerns with the author to see if they agree that there is a problem or not.

Yet I received this non-response response...

Okay, so obviously they don't agree that there is an error, but they haven't said why. In this case, I recommend re-reading the paper as they suggest, soliciting some other opinions on it (as you are doing), and if you still think there is an error you should write back to them and ask them if they can please explain why your position is incorrect. The mere fact that the paper has been through peer review does not mean that you cannot raise concerns and seek withdrawal, so I agree that this is something of a "non-response response".

Hopefully, the matter can be resolved between you and the authors, but if this proves impossible, then you can certainly write back the the journal editor and seek for the journal to make a decision.

$\endgroup$
2
  • $\begingroup$ Thank you. I offered a bounty for 2)Good suggestion(s) for communicating better (beyond applying NVC, which I probably should have). You've definitely provided good suggestions for communicating better, and your criticism is fair - and matches my self-criticism of having not applied NVC. You effectively demonstrate applying NVC. But maybe you're not familiar with the term and it's a coincidence. $\endgroup$ Aug 28 '21 at 9:41
  • 1
    $\begingroup$ It's just a coincidence (and I just googled NVC to find out what it means). It sound like you have a good grasp of the matter. Good luck with everything. $\endgroup$
    – Ben
    Aug 28 '21 at 10:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.