Is significance more important than absolute value? I have three groups and I'm comparing two graphs in each group using the t-test:
the absolute difference between two graphs in the first group is 34 with a p=0.024. (*)
The absolute difference in the second group is 28 with a p=0.04 (*)
and in the third group the absolute difference is 4 but the p value is=0.010 (**)
How do I explain this data? The absolute value in the first and second group is higher; however, the significance is less than the comparison in group three where the absolute value difference between two groups is only 4. So, should I say there was more change in group three (**) than in group 1 and 2 although the absolute difference (4) is less in this group?
Thank you.
 A: The p-value is a nonlinear tranformation of the difference and variation.  This makes it very hard to compare p-values to other p-values meaningfully (other than to say which are significant and which are not significant).
Much better is to compute confidence intervals which show both the absolute difference and the variation around the estimated difference.  While the difference of 34 is the largest, the end of the interval may be closer to 0 than that for the difference of 4 showing that while your estimate is bigger, the variation means that it could in fact be smaller.
A: The P value from the t test depends upon:


*

*The absolute value of the difference between the means

*Variability, as assessed by the SD of the two groups

*Sample size


So the apparent discrepancy in your results must be the result of very different SDs among the groups, or different sample sizes.
Note that the t test assumes that both sets of data are sampled from Gaussian populations with equal standard deviations. 
