Can I compare p-values?

Question

Assume I have 2 vaccines A and B. They are tested on 2 groups of patients. Say each group has n=1000 people drawn from the same population, vaccine A is used on group 1 and vaccine B is used on group 2. It is observed that in group 1, 70 male patients and 40 female patients are cured, while in group 2, 55 male patients and 50 female patients are cured.

I can then test the following hypothesis:

• vaccine A is more effective on male patients than female
• vaccine B is more effective on male patients than female

Each hypothesis test will produce a p-value, p1 and p2. Now I see that p1 is smaller than p2, is there anything I can say regarding which vaccine is more effective on male patients?

Answer 1

You won't be able to tell which effect is larger simply by looking at p values.

You haven't said how many people of each sex were in each group, but let's assume they are evenly split for now.

Let the outcome be $y=0$ for not cured, and $y=1$ for cured. The data are then

 y   Group    Sex   n
1 0 Group A Female 460
2 0 Group A   Male 430
3 0 Group B Female 450
4 0 Group B   Male 445
5 1 Group A Female  40
6 1 Group A   Male  70
7 1 Group B Female  50
8 1 Group B   Male  55

Let's model the risk of being cured using

$$ \log \left( \dfrac{p}{1-p} \right) = \beta_0 + \beta_1x_1 + \beta_2 x_2 + \beta_3x_1x_2 $$

Here:

• $\beta_0$ is the reference category (females from group A).
• $\beta_1$ is the effect of being in group B
• $\beta_2$ is the effect of being male
• $\beta_3$ is the interaction of group B and being male.

Hence, the risk of being cured on the logit scale from group A is

$$\log \left( \dfrac{p}{1-p} \right) = \beta_0 + \beta_2$$

and the risk of being cured on the logit scale from group B is

$$\log \left( \dfrac{p}{1-p} \right) = \beta_0 + \beta_1 + \beta_2 + \beta_3$$

The difference in risks is the quantity $\beta_1 + \beta_3$. Our null is that $\beta_1 + \beta_3 = 0$ versus the alternative that $\beta_1 + \beta_3 \neq 0$. Using R to perform the computations...

# Fit a logistic regression to the data
mod<-glm(y ~ Group*Sex, data = model_data, family=binomial(), weights = n)

# Extract the covariance matrix for the coefficients
Sigma = vcov(model)

# Compute the standard error of the estimate using the covariance matrix

x = c(0, 1, 0, 1)
b = sum(coef(mod) %*% x)
se_b = x %*% Sigma %*% x

# Test statistic
z = b/sqrt(se_b)
>>> -1.43

The test statistic is $z = -1.43$ yielding a p value of about 0.15. We would fail to reject the null hypothesis that males have different risks of being cured across populations.

EDIT:

There may be a simpler way of testing this. The stated hypothesis is really about homogeneity of odds ratios between strata, here stratified by population. We can do this using Cochran's test of homogeneity. I'll work with log odds ratios because they are slightly simpler algebraically.

Let $\hat{\theta}_k$ be the log odds ratio in the $k^{th}$ strata. Let $\hat{\theta}$ be the estimate of the marginal odds ratio (pooling strata). Asymptotically,

$$\widehat{\theta}_{k}-\theta \stackrel{d}{\approx} N\left(0, \sigma_{\widehat{\theta}_{k}}^{2}\right)$$

Let $\tau_k = \sigma^{-2}_{\hat{\theta}_k}$ be the estimated precision of the log odds ratio. A test statistic for homogeneity of odds ratios is

$$X_{H, C}^{2}=\sum_{k} \widehat{\tau}_{k}\left(\widehat{\theta}_{k}-\hat{\theta}\right)^{2}$$

Where $X^2_{H, C} \sim \chi^2_{K-1}$. When I compute this test statistic, I get $X^2_{H, C} = 3.136$ which yields a p value of 0.076, again failing to reject the null. For more, see section 4.6.2 of this book.

Answer 2

Not an elaborated answer since we have already one, but a bit more focus on this part of the question:

Now I see that p1 is smaller than p2, is there anything I can say regarding which vaccine is more effective on male patients?

"The p-value is s the probability of obtaining test results at least as extreme as the results actually observed, under the assumption that the null hypothesis is correct". It is calculated from your sample and used to make inference about the larger population. It is associated to a specific null hypothesis.... and therefore make it complicated to be compared, as such, with another p-value.

Comparing significant with non significant p-value

In the example you provided (and assuming that male and female are equally distributed), the null hypothesis of the second test (vaccine B) cannot be rejected (p2-value = 0.606 for 2-sample test for equality of proportions)... So, a comparison of p-values will be even more than hazardous (if it had made sense), since it is not even significant.

The p-value tells only a little part of the story

Another illustration, the p-value of your first test (vaccine A) is p1-value = 0.0024 (2-sample test for equality of proportions) for 70 male patients and 40 female patients. However, you get a similar p-value of 0.0025 for 399 cured males and 358 cured females.

So, you get the same (or similar) p-value as your initial test for vaccine A, but the overall efficiency of the vaccine is much more important (covering 70% - 80 % of the patients). In such tests, the p-value does not tell you the story of each vaccine and hide their overall efficiency... which can lead to incorrect conclusions.

Below is an illustration of all possible pairs male/female (where vaccine A is more efficient on males) for a very similar p-value as your first test (vaccine A):

In other words, we get similar p-values if vaccine A cures 14 males and 2 females or if it cures 498 males and 486 females.

Finally, it has to tell a story that you can trust, which means that proper analysis must be carried out. Unfortunately, p-values do not tell much about the quality of the analysis: attractive p-values can actually hide poor analyses.

So, there is no other choice than stating a "new" null hypothesis that both vaccines A and B are more effective on male patients than female, as described by @Demetri.

I take advantage of this also to highlight the interesting link (direct access) mentioned by @StephanKolassa in comments (since comments are not always read): "The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant" - Gelman & Stern (2006).

Answer 3

Since effect size is one factor going into p-values, there are cases where p-values tell you something useful about effect sizes, but just looking at effect sizes directly is always more useful.

You have the further issue that your p-values are from testing the hypothesis of the vaccine being more effective on males than females, rather than directly testing effectiveness on males. This means that your p-values are differing not only from the variable you care about (effectiveness on males) but also on one you don't (effectiveness on females). This makes the p-values even closer to useless.

There would be little reason to not simply do hypothesis testing based on a binomial or Gaussian hypothesis of the data. If for some reason you were only given the p-values and did not have access to the raw data, the best course of action would probably be to simply refrain from trying to draw conclusions.

As as a purely medical, rather than statistical note, generally vaccines are used to prevent infection rather than cure them (although I suppose there are cases where they can do the latter).