What is the probability that the Pfizer/BioNTech vaccine is not/less effective than the study suggests?

Question

The Pfizer/BioNTech vaccine has a reported efficacy of 90%. AFAIK, the phase III study has the following preliminary results (as of November 9):

• 43'000 people were given a shot, 50% received the vaccine, 50% a placebo
• Of these 43'000 people, 94 got tested positive for Covid-19 during the study
• Of these 94 people, fewer than nine were in the group that received the vaccine, the others were in the control group.
• This means we have a (preliminary) efficacy of more than 90%.

Let's give these numbers some variable names (and let's fix some numbers to get rid of the "fewer than"):

\begin{aligned}[ll] N &= 43'000\\N_{vac}&=21'500\\N_{cg}&=21'500\\C_{vac}&=9\\C_{cg} &= 85 \end{aligned}

Both the vaccinated group and the control group have 21'500 members. There are $C_{vac}=9$ Covid-19 diagnoses within the vaccinated group, and $C_{cg}=85$ in the control group.

I would like to use these numbers to calculate the probability $P_X$ that the vaccine has efficacy $X$ or higher.

Of course, $P_X$ should have its steepest part around $X=90\%$, because that is the most likely efficacy of the vaccine, according to the study. But still, we have the possibility that the study over-estimated the efficacy of the vaccine, because of random effects.

The higher $N$ is, the more "narrow" we expect the derivative of $P_X$ to be (the more participants in the study, the less uncertainty). For very small $N$, $P_X$ should be very close to $X$ itself, because we do not know much about the vaccine.

How can I calculate $P_X$?

Answer 1

They used a Beta-Binomial Bayesian model to calculate the posterior efficacy (link to study plan as PDF).

First, letting $\pi_v$ and $\pi_c$ be the population probabilities that a vaccinated or a control subject contract Covid-19, respectively. The population vaccine efficacy is defined by $$ \mathrm{VE}=1 - \frac{\pi_v}{\pi_c} $$

The statisticians assume a prior distribution for the parameter $$ \theta = \frac{1 - \mathrm{VE}}{2 - \mathrm{VE}} $$

Substituting the definition of the efficacy, this can be rewritten as $$ \theta = \frac{\pi_v}{\pi_v + \pi_c} $$ So $\theta$ is the probability that a subject who contracted Covid-19 is from the vaccinated group and $1-\theta$ is the probability that the subject was in the control group.

The study plan states that they assumed a Beta($a_0 =0.700102, b_0=1$) prior for $\theta$ which results in a prior mean of $\mathrm{E}(\theta)=0.4118$ which corresponds to a prior mean of 30% efficacy, because $\mathrm{VE}=(1 - 2\theta)/(1-\theta)$.

Recall that the posterior is simply another Beta distribution with parameters $a_1 = a_0 + m_v$ and $b_1 = b_0 + m_c$ where $m_v$ and $m_c$ denote the number of subjects that fell ill in the vaccinated and control group, respectively. You can calculate all desired probabilities from this posterior distribution (and updated it when more data are available).

Let's illustrate it with the data given in your post. We have $m_v = 9$ and $m_c = 85$ so the posterior Beta distribution has parameters $a_1 = 0.700102 + 9 = 9.700102$ and $b_1 = 1 + 85 = 86$. We are interested in the probability that the vaccine has 90% efficacy or more which corresponds to a $\theta \leq 1/11$.

The calculations in R are as follows:

# Priors
a0 = 0.700102
b0 = 1

# Posterior
a1 = a0 + 9
b1 = b0 + 85

# Critical probability
p_crit <- 0.9
theta_crit <- (p_crit - 1)/(p_crit - 2)

# Posterior probability
pbeta(theta_crit, a1, b1)
[1] 0.3982548

# Mode
theta_mode <- (a1 - 1)/(a1 + b1 - 2)
(1 - 2*theta_mode)/(1 - theta_mode)
[1] 0.8976459

So the posterior probability that the efficacy is greater than 90% is $0.398$ which corresponds to the grey area in the plot. The mode of the posterior Beta distribution is at $0.093$ which translates to an efficacy of $0.898$, which corresponds nicely to the reported efficacy of 90%.

Here is the plot of the posterior density. The red vertical line denotes the $\theta$ below which the efficacy is $\geq$90% (depicted by the grey area):

A more detailled exposition can be found in this blog.