# How is the vaccine effect calculated?

I'm trying to understand this study about the vaccine effect for the COVID. In the results the authors state:

We found that VEs increased over time until a peak at day 28-34 days post-vaccination for both vaccines. The highest VE against COVID-19 hospitalisation amongst those receiving the first dose of the vaccine BNT162b2 was 85%, (95% CI 76 to 91) and for ChAdOx1 it was 94% (95% CI 73 to 99)

This refers to table 2 were you can find the data mentioned:

How have the authors arrived to the 85%/94% of vaccine effect? And what does it means in the real world? Because what I see is that a 0'53%/0'38% of vaccinated people still go to hospital vs a 0'52% of non-vaccinated. What I'm missing?

• The number of cases and the person time at risk in the unvaccinated group is needed in order to estimated VE. I don't see these informations in the table. Feb 23, 2021 at 12:17
• Do you mean you can't find it in the resume of the table I've post or in the linked paper? The complete table is in the linked paper (table 2). The data this table gives for the unvaccinated is: Overall: Unvaccinated 787518 7472 Pfizer: Unvaccinated 708129 6690 AstraZeneca: Unvaccinated 700859 7090
– Ivan
Feb 23, 2021 at 18:28

The CDC website describes in general how vaccine efficacy/effectivenss is measured.

https://www.cdc.gov/csels/dsepd/ss1978/lesson3/section6.html

"Vaccine efficacy/effectiveness (VE) is measured by calculating the risk of disease among vaccinated and unvaccinated persons and determining the percentage reduction in risk of disease among vaccinated persons relative to unvaccinated persons."

The formula is 1 minus estimated relative risk, expressed as a percentage.

Thus, the easy answer to your question using the data from the cited paper is as follows:

“the VE (vaccine effect) is calculated as 1 minus the full and inverse propensity adjusted Hazard Ratio (an estimate of relative risk) shown in column 6 expressed as a percentage. Thus, for vaccine 1, the full inverse propensity adjusted Hazard Ratio in the period 28-34 days after vaccination is 0.15 and thus the VE is 1 minus 0.15 or 0.85, which is 85% when expressed as a percentage.”

If this study were a randomized trial, one could estimate the relative risk of hospitalization in the vaccinated by dividing the rate of hospitalization in the vaccinated in the interval 28-34 days after vaccination by the rate of hospitalization in the unvaccinated. The vaccine effect estimate would be 1 minus the relative risk expressed as a percentage. Using the data in Table 2 for vaccine 1, the rate of hospitalization in the vaccinated in the interval 28-34 days after vaccination is 18/3,842 or 5.4 per 1,000 person-years. The rate of hospitalization in the unvaccinated is 6,690/708,129 or 9.4 per 1,000 person-years. The crude relative risk of hospitalization in the unvaccinated compared with the vaccinated is 5.4/9.4 or 0.58. Using this estimate of the crude relative risk, the VE (vaccine effect) would be 42% (1 minus 0.58 expressed as a percentage).

But the data are not from a randomized trial. The crude estimate of relative risk does not account for differences in age (and sex and co-morbidity and socioeconomic status and other things) between the vaccinated and unvaccinated that might affect the risk of hospitalization. Some of these differences are large (Table 1). There is uncontrolled confounding in an analysis using the crude data. Especially important differences between the vaccinated and unvaccinated are age, sex, number of co-morbidities and high blood pressure. These are all related to hospitalization in prior epidemiologic studies. Secular time is also a factor of concern because the rates of hospitalization may be changing given that the study encompasses a pandemic period. Time must be accounted for in the analysis.

To account for confounding and secular time, the authors did a multivariate analysis that is described in detail in the methods section.

The results of the multivariate analysis are shown in columns 4, 5, and 6 of Table 2. The authors first adjusted only for age (results are in column 4). They then adjusted for time (in weeks), age, sex, Scottish Index of Multiple Deprivation (SIMD), number of RT-PCR tests prior to vaccination and number of underlying medical conditions (results are in column 5). Finally, the authors did a propensity score analysis that used inverse propensity score weighting with adjustment for time (in weeks), age, sex, Scottish Index of Multiple Deprivation (SIMD), number of RT-PCR tests prior to vaccination and number of underlying medical conditions (results are in column 6).

A recent (2019) published review by Ali describes propensity methods in detail.

https://www.frontiersin.org/articles/10.3389/fphar.2019.00973/full

Ali MS, Prieto-Alhambra D, Lopes LC. Propensity Score Methods in Health Technology Assessment: Principles, Extended Applications, and Recent Advances. Front Pharmacol. 2019 Sep 18;10:973. doi: 10.3389/fphar.2019.00973. PMID: 31619986; PMCID: PMC6760465.

There is much useful information about propensity methods in posts at the Cross-Validated site and tagged as “propensity score.”

Clearly, the adjustments and the analysis approach affect the conclusions. The age-adjusted Hazard Ratio for the interval 28-34 days after vaccination is 0.33, the fully adjusted Hazard Ratio for the interval 28-34 days after vaccination is 0.22, and the full and inverse propensity adjusted Hazard Ratio is 0.15. These Hazard Ratios translate to the following estimates of the vaccine effect for the interval 28-34 days after vaccination: age-adjusted 67% (1 minus 0.33), fully adjusted 78% (1 minus 0.22), full and propensity adjusted 85% (1 minus 0.15).

If one believes that the “best” estimate of the effect of vaccine in reducing the likelihood of being hospitalized comes from the full and inverse propensity weighted analysis, the conclusion is that vaccine 1 reduces the chances of being hospitalized by a factor of 0.15 (on average). Put another way, the chances of being hospitalized are 6.7 times higher in unvaccinated than in vaccinated people (1/0.15). This is a very large effect size.

If one does not accept the full and inverse propensity weighted analysis and bases conclusions on the measure of vaccine effectiveness that uses the age-adjusted Hazard Ratio (0.33—vaccine effect 67% reduction) or the fully adjusted Hazard Ratio (0.22—vaccine effect 78%), the effect size is less impressive.

With such large differences in factors that affect hospitalization comparing the vaccinated and unvaccinated, no conclusion about the vaccine effect can be drawn from the crude data.

The absolute effect of vaccination on hospitalization would depend on the underlying chances of hospitalization (and an assumption that the propensity adjusted hazard ratio is uniformly applicable at every level of risk of hospitalization). Estimating this for various subgroups or for an individual over time would require a model (and probably more data)

• Wow! Excellent answer. Thanks. I've learnt a lot about epidemiologic statistics. But, why they don't make two equivalent groups vaccinated and unvaccinated with similar variables (age, co-morbidities, sex, etc) so they can just compare using standard methods? Is it because ethical issues?
– Ivan
Feb 26, 2021 at 11:59
• @Ivan Advocates of propensity score methods consider them to be less affected by "model misspecification" compared with traditional regression analysis. They have become "trendy" especially in pharmacoepidemiology. Material in Cross-Validated with the tag "propensity scores" by Noah contains many references and a very carefully thought out discussion of the pros and cons. Feb 26, 2021 at 17:25