Analyzing Pfizer Vaccine Efficacy: Testing a Claim about 2 Proportions I teach Introduction to Statistics and would like to show my students how to analyze the Pfizer Phase 3 Vaccine results.  Testing a claim about 2 proportions is straightforward with the Pfizer data:
Given:

*

*Group (1) is the placebo group;

*Group (2) is the vaccine group

Data:

*

*https://www.pfizer.com/news/press-release/press-release-detail/pfizer-and-biontech-conclude-phase-3-study-covid-19-vaccine [A], and


*https://pfe-pfizercom-d8-prod.s3.amazonaws.com/2020-11/C4591001_Clinical_Protocol_Nov2020.pdf [B]
The number of people who got the 2nd dose of vaccine is 41,135 [A].  Assuming that people dropped out equally from the placebo and vaccine group, those groups were created at a 1:1 ratio [B].  So we have approximately 20,567 people in each group.  Using the number of people who got covid from each group published in [A] we have:
Placebo Group:   $x_1 = 162$,   $n_1 = 20,567$
Vaccine Group (BNT162b2):  $x_2 = 8$,  $n_2 = 20,567$
Claim:  $p_1 > p_2 \rightarrow p_1 - p_2 > 0$
$H_0: p_1 - p_2 \le 0$ 
$H_1: p1 - p2 > 0$
Test Statistic: $z = 11.8  \rightarrow \text{P-Value} < 0.0001$
Thus we have evidence for the claim that $p_1 > p_2$ meaning that we have evidence for the claim that the vaccine lowered the covid rate versus the control group.
So far, so good.  But here's where things break down.  The manufacturer's claim that the vaccine is shown to be 95% effective.  If I use the 162 as the expected number of cases in general, then if the vaccine is 95% effective, the vaccine group should have less than 5% of the expected number of cases.  5% of 162 is 8.1 cases expected in 20567.
Control Group:  $x_1 = 8.1, n_1 = 20,567$
Experimental Group:  $x_2 = 8, n_2 = 20,567$
Claim:  $p_1 > p_2  \rightarrow p_1 - p_2 > 0$
$H0: p_1 - p_2 \le 0$ 
$H1: p_1 - p_2 > 0$
Test Statistic: $0.0249 \rightarrow  \text{P-Value} = 0.4901.$
This is clearly a fail meaning that with a P-Value of 0.4901, I have NOT shown that the vaccine group has less than 5% of the expected number of cases.  Why is this analysis flawed?  This seems related to (What does 94.5% effective mean?), but even reading this citation, its unclear to me why my hypothesis test is wrong.  Again, an explanation appropriate for an INTRODUCTION to statistics class, please.

UPDATE:  If I redo the analysis with an expectation of 16.2 cases instead of 8.1, then I get a p-value of 0.0477.  This suggests that I can claim with a 5% significance level that the vaccine is 90% effective.  Is the issue just that they can do a more detailed/sophisticated analysis to get to 95% effective or is my analysis just flawed and I've gotten lucky?
 A: Based on the comments, there are several issues with this analysis.  First, the analysis is based on being able to approximate binomial as normal.  Textbooks I've consulted write this requirement in different ways. Sullivan (Fundamentals of Statistics) says that in addition to being binomial, each of the groups must pass $np(1-p) \ge 10$ for this to be valid.  For the control group this requirement is satisfied.  For the experimental group it is not.    Triola (Elementary Statistics) says the requirement is  $np\ge5$ and $nq\ge 5$ for each of the two groups.  This requirement is met for both groups.
Second, the second analysis also depends on the control group being an estimate of the number of cases expected in the experimental group.  This assumption is inaccurate and could cause the vaccine to appear less effective than it is.
Finally, for the confidence interval, again the textbooks give differing requirements ($np(1-p) \ge 10$ vs $np \ge 5$ and $nq \ge 5$).  Only some of these requirements are met, suggesting we are on the hairy edge of validity with this analysis.  So a crude guess about the proportion who got covid is ($1.6\%$ to $8.2\%$).
What all of the means to me is that we can do some basic analysis, but this has to be taken with a grain of salt because the number of cases is still sufficiently small to not robustly meet the requirements.
Finally, thanks to the people who commented on this posting as they helped me to think more deeply about what was going on
