# Can I compare p-values?

Assume I have 2 vaccines A and B. They are tested on 2 groups of patients. Say each group has n=1000 people drawn from the same population, vaccine A is used on group 1 and vaccine B is used on group 2. It is observed that in group 1, 70 male patients and 40 female patients are cured, while in group 2, 55 male patients and 50 female patients are cured.

I can then test the following hypothesis:

• vaccine A is more effective on male patients than female
• vaccine B is more effective on male patients than female

Each hypothesis test will produce a p-value, p1 and p2. Now I see that p1 is smaller than p2, is there anything I can say regarding which vaccine is more effective on male patients?

• I am not aware of any useful relations between effect sizes and p-values, but I would certainly be happy to learn of new proven propositions. Alas, you are probably better off comparing effect sizes rather than p-values. Nov 2 '21 at 3:47
• Remember that p-values are themselves random variables with variability, and that the values you see are just point estimates, which you can't really deduce more from than from the point estimates of vaccine effectiveness in your groups. Gelman & Stern (2006) may also be informative. Nov 2 '21 at 5:46
• @StephanKolassa Excellent point about the p-value being a random variable. Taleb looks at deriving a metadistribution. Nov 2 '21 at 15:37
• @Galen Given that there is a formula for finding p-values given effect size and sample size, you can, given p-value and sample size, solve for effect size. Of course, "you can" here has a meaning similar to its meaning in such sentences as "If you have a nail you need driven into a board, you can use your head as a hammer." Nov 3 '21 at 0:39
• @Acccumulation Yes, if you additionally include sample size there are formulae for some tests, such as for a Student's t test. I am aware of these, although they are not always available. Sometimes Monte Carlo methods can be used assuming a data generating process. My comment was intended to the case of 'only' effect size and p-values, but I appreciate you introducing more nuance. The need for your comment suggests my phrasing was simplistic. Nov 3 '21 at 0:58

## 3 Answers

You won't be able to tell which effect is larger simply by looking at p values.

You haven't said how many people of each sex were in each group, but let's assume they are evenly split for now.

Let the outcome be $$y=0$$ for not cured, and $$y=1$$ for cured. The data are then

 y   Group    Sex   n
1 0 Group A Female 460
2 0 Group A   Male 430
3 0 Group B Female 450
4 0 Group B   Male 445
5 1 Group A Female  40
6 1 Group A   Male  70
7 1 Group B Female  50
8 1 Group B   Male  55


Let's model the risk of being cured using

$$\log \left( \dfrac{p}{1-p} \right) = \beta_0 + \beta_1x_1 + \beta_2 x_2 + \beta_3x_1x_2$$

Here:

• $$\beta_0$$ is the reference category (females from group A).
• $$\beta_1$$ is the effect of being in group B
• $$\beta_2$$ is the effect of being male
• $$\beta_3$$ is the interaction of group B and being male.

Hence, the risk of being cured on the logit scale from group A is

$$\log \left( \dfrac{p}{1-p} \right) = \beta_0 + \beta_2$$

and the risk of being cured on the logit scale from group B is

$$\log \left( \dfrac{p}{1-p} \right) = \beta_0 + \beta_1 + \beta_2 + \beta_3$$

The difference in risks is the quantity $$\beta_1 + \beta_3$$. Our null is that $$\beta_1 + \beta_3 = 0$$ versus the alternative that $$\beta_1 + \beta_3 \neq 0$$. Using R to perform the computations...

# Fit a logistic regression to the data
mod<-glm(y ~ Group*Sex, data = model_data, family=binomial(), weights = n)

# Extract the covariance matrix for the coefficients
Sigma = vcov(model)

# Compute the standard error of the estimate using the covariance matrix

x = c(0, 1, 0, 1)
b = sum(coef(mod) %*% x)
se_b = x %*% Sigma %*% x

# Test statistic
z = b/sqrt(se_b)
>>> -1.43


The test statistic is $$z = -1.43$$ yielding a p value of about 0.15. We would fail to reject the null hypothesis that males have different risks of being cured across populations.

EDIT:

There may be a simpler way of testing this. The stated hypothesis is really about homogeneity of odds ratios between strata, here stratified by population. We can do this using Cochran's test of homogeneity. I'll work with log odds ratios because they are slightly simpler algebraically.

Let $$\hat{\theta}_k$$ be the log odds ratio in the $$k^{th}$$ strata. Let $$\hat{\theta}$$ be the estimate of the marginal odds ratio (pooling strata). Asymptotically,

$$\widehat{\theta}_{k}-\theta \stackrel{d}{\approx} N\left(0, \sigma_{\widehat{\theta}_{k}}^{2}\right)$$

Let $$\tau_k = \sigma^{-2}_{\hat{\theta}_k}$$ be the estimated precision of the log odds ratio. A test statistic for homogeneity of odds ratios is

$$X_{H, C}^{2}=\sum_{k} \widehat{\tau}_{k}\left(\widehat{\theta}_{k}-\hat{\theta}\right)^{2}$$

Where $$X^2_{H, C} \sim \chi^2_{K-1}$$. When I compute this test statistic, I get $$X^2_{H, C} = 3.136$$ which yields a p value of 0.076, again failing to reject the null. For more, see section 4.6.2 of this book.

• Just knowing the N in each experimental group split by gender, and the p-values should be sufficient, at least to a very good approximation --- you can back out the T statistics via an inverse transform and use the central limit theorem, assuming gender and treatment assignment are orthogonal if necessary
– JDL
Nov 2 '21 at 14:33
• @JDL No, because the p values are against the null of no effect, and the stated hypothesis is about homogeneity between groups. Its a slightly different question altogether. Nov 2 '21 at 14:41
• sure, but once we have recovered beta-1-hat and beta-2-hat, we can use those to test the hypothesis of beta-1 = beta-2 even if they were originally intended for testing against beta-1=0 and beta-2=0 separately?
– JDL
Nov 2 '21 at 14:43
• @JDL and suppose one p value is very large and the other very small? What do we conclude then? This may work in ideal cases, but I'd prefer we follow the more general rule of "don't use p values to determine effect sizes". Nov 2 '21 at 14:50
• it would depend on the sample sizes. Say p1 was 0.00003, corresponding to a 4/sqrt(n1) sigma effect and p2 was 0.69, corresponding to a -0.5/sqrt(n2) sigma effect. Then the difference between the two can be computed and its significance assessed depending on what n1 and n2 are.
– JDL
Nov 2 '21 at 15:12

Not an elaborated answer since we have already one, but a bit more focus on this part of the question:

Now I see that p1 is smaller than p2, is there anything I can say regarding which vaccine is more effective on male patients?

"The p-value is s the probability of obtaining test results at least as extreme as the results actually observed, under the assumption that the null hypothesis is correct". It is calculated from your sample and used to make inference about the larger population. It is associated to a specific null hypothesis.... and therefore make it complicated to be compared, as such, with another p-value.

Comparing significant with non significant p-value

In the example you provided (and assuming that male and female are equally distributed), the null hypothesis of the second test (vaccine B) cannot be rejected (p2-value = 0.606 for 2-sample test for equality of proportions)... So, a comparison of p-values will be even more than hazardous (if it had made sense), since it is not even significant.

The p-value tells only a little part of the story

Another illustration, the p-value of your first test (vaccine A) is p1-value = 0.0024 (2-sample test for equality of proportions) for 70 male patients and 40 female patients. However, you get a similar p-value of 0.0025 for 399 cured males and 358 cured females.

So, you get the same (or similar) p-value as your initial test for vaccine A, but the overall efficiency of the vaccine is much more important (covering 70% - 80 % of the patients). In such tests, the p-value does not tell you the story of each vaccine and hide their overall efficiency... which can lead to incorrect conclusions.

Below is an illustration of all possible pairs male/female (where vaccine A is more efficient on males) for a very similar p-value as your first test (vaccine A):

In other words, we get similar p-values if vaccine A cures 14 males and 2 females or if it cures 498 males and 486 females.

Finally, it has to tell a story that you can trust, which means that proper analysis must be carried out. Unfortunately, p-values do not tell much about the quality of the analysis: attractive p-values can actually hide poor analyses.

So, there is no other choice than stating a "new" null hypothesis that both vaccines A and B are more effective on male patients than female, as described by @Demetri.

I take advantage of this also to highlight the interesting link (direct access) mentioned by @StephanKolassa in comments (since comments are not always read): "The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant" - Gelman & Stern (2006).

Since effect size is one factor going into p-values, there are cases where p-values tell you something useful about effect sizes, but just looking at effect sizes directly is always more useful.

You have the further issue that your p-values are from testing the hypothesis of the vaccine being more effective on males than females, rather than directly testing effectiveness on males. This means that your p-values are differing not only from the variable you care about (effectiveness on males) but also on one you don't (effectiveness on females). This makes the p-values even closer to useless.

There would be little reason to not simply do hypothesis testing based on a binomial or Gaussian hypothesis of the data. If for some reason you were only given the p-values and did not have access to the raw data, the best course of action would probably be to simply refrain from trying to draw conclusions.

As as a purely medical, rather than statistical note, generally vaccines are used to prevent infection rather than cure them (although I suppose there are cases where they can do the latter).