Let's say you have A, B and C data sets. You're running t-tests on them and let's assume the p-value between A and B is 0.02 and between A and C is 0.0004. Does it mean that A is more different from C than is from B?
4
-
$\begingroup$ That's the tricky question: as I assume you estimate variances from data and moreover you estimate pooled variance. But if the variances of A,B,C are equal and the sizes of groups are equal - p-value of t-test is monotonically dependent on the effect size, if I am not mistaken, and the answer is "yes". If the sizes of groups are unequal - no (other assumptions are ± avoidable). $\endgroup$– German DemidovCommented Nov 20, 2019 at 10:50
-
$\begingroup$ Actually my comment is just a periphrase of stats.stackexchange.com/questions/21419/… $\endgroup$– German DemidovCommented Nov 20, 2019 at 10:57
-
$\begingroup$ You are doing multiple testing so should perhaps be using something like an ANOVA. $\endgroup$– Paul HewsonCommented Nov 20, 2019 at 11:00
-
$\begingroup$ Under the conditions posited, $A$ and $C$ could be nearly identical; they certainly could have closer means than $A$ and $B$ do, because the p-value depends strongly on sample size and variance, not just on the means that you are comparing. $\endgroup$– whuber ♦Commented Nov 20, 2019 at 15:33
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
No. For a variety of reasons. First, $p$-values are random numbers and depending on cirumstances $p_1$ = 0.02 and $p_2$ = 0.0004 may stem from the same distribution. If $H_0$ was true (which it almost never is), then $p$=0.02 and $p$=0.0004 are equally likely values. If you want to compare in-sample, just take the means of the samples and compare them. If you want to compare out-of-sample $p$-values are not a suitable means for that (nobody ever tests for significance of p-value differences).
Second: Even if the true means of A-and-B and A-and-C differ by the same amount, the resulting $p$-values will also depend on the variances/standard deviations (including measurement precision) of A, B and C. As t-values depend on differences of means and on variances, they are nor a measure of differences of means alons.
$\begingroup$
$\endgroup$
$p$-value of a $t$-test is just transformation of $t$-statistics into probability so the same question can be answered by answering question if $t$-statistics can tell you A is more different from C. Since A, B and C are different datasets you have to use independent samples $t$-test, which for A vs B is:
$$t=\frac{\bar{x}_A-\bar{x}_B}{\sqrt{\frac{s_A^2}{n_A}+\frac{s_B^2}{n_B}}}$$
If by different you mean having different sample means then answer is no in general, yes only in special cases. The $t$-statistics and $p$-value, save for special cases, won’t be informative about differences in mean because the number of observations and value of variance changes between A, B and C. Even if by different you mean exact shape of the assumed $t$ distribution again you will have a problem to say if they are more different if the number of observations is not kept constant and you won’t be able to say if the difference comes more from mean or standard deviation just from the result. However, if you have equal variances and number of observations then answer would be yes.
