What sense does it make to compare p-values to each other?

Question

I have two populations (men and women), each containing $1000$ samples. For each sample I have two properties A & B (first year grade point average, and SAT score). I have used a t-test separately for A & B: both found significant differences between the two groups; A with $p=0.008$ and B with $p=0.002$.

Is it okay to claim that the property B is better discriminated (more significant) then the property A? Or is it that a t-test is just a yes or no (significant or not significant) measure?

Update: according to the comments here and to what I have read on wikipedia, I think that the answer should be: drop the meaningless p-value and report your effect size. Any thoughts?

Answer 1

Many people would argue that a $p$-value can either be significant ($p< \alpha$) or not, and so it does not (ever) make sense to compare two $p$-values between each other. This is wrong; in some cases it does.

In your particular case there is absolutely no doubt that you can directly compare the $p$-values. If the sample size is fixed ($n=1000$), then $p$-values are monotonically related to $t$-values, which are in turn monotonically related to the effect size as measured by Cohen's $d$. Specifically, $d=2t/\sqrt{n}$. This means that your $p$-values are in one-to-one correspondence with the effect size, and so you can be sure that if the $p$-value for property A is larger than for property B, then the effect size for A is smaller than for property B.

I believe this answers your question.

Several additional points:

1. This is only true given that the sample size $n$ is fixed. If you get $p=0.008$ for property A in one experiment with one sample size, and $p=0.002$ for property B in another experiment with another sample size, it is more difficult to compare them.

   • If the question is specifically whether A or B are better "discriminated" in the population (i.e.: how well can you predict gender by looking at the A or B values?), then you should be looking at effect size. In the simple cases, knowing $p$ and $n$ is enough to compute the effect size.

   • If the question is more vague: what experiment provides more "evidence" against the null? (this can be meaningful if e.g. A=B) -- then the issue becomes complicated and contentious, but I would say that the $p$-value by definition is a scalar summary of the evidence against the null, so the lower the $p$-value, the stronger the evidence, even if the sample sizes are different.

2. Saying that the effect size for B is larger than for A, does not mean that it is significantly larger. You need some direct comparison between A and B to make such a claim.

3. It's always a good idea to report (and interpret) effect sizes and confidence intervals in addition to $p$-values.

Answer 2

Thanks to whoever just downvoted me, as I now have a completely different answer to this question.I have accordingly deleted my original answer as it is incorrect from this perspective.

In the context of this question, which is only dealing with the question "was A or B a better discriminator in my study", we are dealing with a census and not a sample. Thus, the use of inferential statistics such as those used to produce p-values are irrelevant. Inferential statistics are used to infer population estimates from those we obtain from our sample. If we do not wish to generalise to a population, then those methods are unnecessary. (There are some specific issues around missing values in a census, but those are irrelevant in this situation.)

There is no probability of obtaining a result in a population. We obtained the result we got. Therefore, the probability of our results is 100%. There is no need to construct a confidence interval - the point estimate for the sample is exact. We're simply not having to estimate anything at all.

In the specific case of "which variable worked better with the data I have", all one needs to do is look at the results in simple summary form. A table may be sufficient, maybe a graph like a box plot.

Answer 3

You get a difference in p, but it is unclear what that difference means (is it large, small, significant?)

Maybe use bootstrapping:

select (with replacement) from your data, redo your tests, compute difference of p's (p_a - p_b) , repeat 100-200 times

check what fraction of your delta p's is < 0 (meaning p of A is below p of B)

Note: I have seen this done, but am not an expert.

Answer 4

Added an answer as it was too long for a comment!

Michelle has a good response, but the many comments show some common discussions that come up about p-values. The basic ideas are the following:

1) A smaller p-value doesn't mean a result is more or less significant. It just means that the chances of getting a result at least as extreme are less likely. Significance is a binary outcome based on your chosen significance level (which you choose before you run the test).

2) Effect size (often standardized to #'s of standard deviations) is a good way to quantify "how different" two numbers are. So if Quantity A has an effect size of .8 standard deviations and Quantity B has an effect size of .5 standard deviations, you'd say that there's a larger difference between the two groups in Quantity A than in Quantity B. Standard measurements are:

.2 standard deviations = "small" effect

.5 standard deviations = "medium" effect

.8 standard deviations = "large" effect