Significant differences and related variables

Question

I am reading a paper in which a statistical difference has been found between two treatment groups for a variable A (p=.05), but not for variables B (p=.06) and C(p=.06), even though A=B-C. Is this possible, and if so, how? The paper states that "All data were statistically analyzed using the Student's independent t-test provided the data were normally distributed. Otherwise, the Wilcoxon-two-sample test was used."

Answer 1

The mathematical equation (A = B - C) sounds completely unrelated to the tests described in the question.

The t-tests are significant by an arbitrary cutoff. In this case it sounds like 0.05 was selected. That's a decision about what kind of meaning can be ascribed to effects found but is not a measure of the actual size of the effects. Furthermore, there is no test done of the difference between A and B (or C). That's a separate thing. So, while the t-test used would allow one to claim the levels of A differ but we're undecided on the levels of B, it doesn't say anything about differences between A and B effects. The difference between significant and not-significant would very likely not be significant in this case but should be directly tested.

If the paper is about extracting a lot of meaning from difference between the separate tests of A and B without testing then the authors were in error. This may not be serious if the effects are in different directions. But, if they are all in the same direction then there's probably nothing to the significant and not-significant distinction.

Gelman, A. (2006). The difference between “significant” and “not significant” is not itself statistically significant. The American Statistician, 60(4):328–331.

Answer 2

That is possible. Imagine you are looking into the growth of your savings. These are related to your income minus your expenses (A-B) but taken on their own, neither income(A) nor expenses(B) can predict your savings.

Besides, keep in mind that p=0.05 is a widely used convention but still an arbitrary number. So p=0.06 is 'nearly significant' while 0.05 is 'barely significant'.

Answer 3

I can't contrive a situation or simulation where B and C are completely unrelated to Y but A=B-C is. However, it is easy to set up a situation, where Y is more strongly related to A than it is to either B or C. These situations depend on B and C somehow being related to eachother, as in @Henk's example of B being income and C expenditure, hence A is savings.

Consider the situation below. Y is quite strongly related to A (correlation > 0.3) and only weakly to B and C (correlation <0.1), even though A=B-C. The secret is, roughly interpreted, that B and C almost cancel eachother out. The R code that generated these data is:

VarB <- rnorm(1000)
VarC <- VarB*(rnorm(1000,1,.1))
VarA <- VarB-VarC
Y <- 10 + 3 *VarA + rnorm(1000)

I think it likely that you have a similar situation in your case. I am sure your response variable is in fact related to each of A, B and C. The distinction between p values of 0.05 and 0.06 is not meaningful - have a search through this and other sites on the shortcomings of hypothesis testing.

So basically what you have come up with here is a reason for not putting any weight on the arbitrary use of 0.05 as a cut-off point for "significant" results. A much better approach might be to look for results where there is statistically significant evidence that the relationship is material (I prefer to use "material" for this purpose to avoid annoying linguistic paradoxes about the word "significant").