Skip to main content
Plot added to illustrate the problem
Source Link
Ben M.
  • 205
  • 3
  • 10

Now that I've run a couple of tests in R, I think I understand the nature of the problem a little better, so I'll make an attempt at answering my own question. To rephrase the original problem, basically the test of interest boils down to comparing an association between two independent groups. The way I see it there are three approaches to this comparison:

  1. Fisher's test for comparing independent correlations
  2. Regressing $Y$ on $X$ with a grouping variable $G$ and testing the interaction (as above)
  3. Regressing $X$ on $Y$ with a grouping variable $G$ and testing the interaction (as above)

Conceptually, all three tests ask whether some $XY$ association is significantly different in one group versus another. Having run many examples, however, I found that these three tests do not always agree, and sometimes even diverge dramatically. This seems to be something of a paradox to me.

Approaches 2 and 3 will only produce identical results if the $XY$ data are standardized within each group, prior to running the regression. This makes the regression analysis somewhat similar to Fisher's test (to my intuition, at least). For unstandardized data, the two regression tests usually agree but diverge more depending on two conditions:

  1. Larger $Y$ variance in one group versus another
  2. Moderate divergence of $XY$ correlation in the two groups

From what I've run, condition 1 seems to have an impact only to the extent that condition 2 is also an issue. When the two tests diverge strongly, one of the two models usually shows heteroscedasticity problems. The fastest way to spot something is wrong is by comparing the regression output to Fisher's test, though. I have added a picture of some data below, where the slopes differ dramatically between groups depending on whether you regress on $X$ or $Y$.

If someone could weigh in with some theoretical insight it would be most appreciated!

Dramatic difference between regression tests

Now that I've run a couple of tests in R, I think I understand the nature of the problem a little better, so I'll make an attempt at answering my own question. To rephrase the original problem, basically the test of interest boils down to comparing an association between two independent groups. The way I see it there are three approaches to this comparison:

  1. Fisher's test for comparing independent correlations
  2. Regressing $Y$ on $X$ with a grouping variable $G$ and testing the interaction (as above)
  3. Regressing $X$ on $Y$ with a grouping variable $G$ and testing the interaction (as above)

Conceptually, all three tests ask whether some $XY$ association is significantly different in one group versus another. Having run many examples, however, I found that these three tests do not always agree, and sometimes even diverge dramatically. This seems to be something of a paradox to me.

Approaches 2 and 3 will only produce identical results if the $XY$ data are standardized within each group, prior to running the regression. This makes the regression analysis somewhat similar to Fisher's test (to my intuition, at least). For unstandardized data, the two regression tests usually agree but diverge more depending on two conditions:

  1. Larger $Y$ variance in one group versus another
  2. Moderate divergence of $XY$ correlation in the two groups

From what I've run, condition 1 seems to have an impact only to the extent that condition 2 is also an issue. When the two tests diverge strongly, one of the two models usually shows heteroscedasticity problems. The fastest way to spot something is wrong is by comparing the regression output to Fisher's test, though.

If someone could weigh in with some theoretical insight it would be most appreciated!

Now that I've run a couple of tests in R, I think I understand the nature of the problem a little better, so I'll make an attempt at answering my own question. To rephrase the original problem, basically the test of interest boils down to comparing an association between two independent groups. The way I see it there are three approaches to this comparison:

  1. Fisher's test for comparing independent correlations
  2. Regressing $Y$ on $X$ with a grouping variable $G$ and testing the interaction (as above)
  3. Regressing $X$ on $Y$ with a grouping variable $G$ and testing the interaction (as above)

Conceptually, all three tests ask whether some $XY$ association is significantly different in one group versus another. Having run many examples, however, I found that these three tests do not always agree, and sometimes even diverge dramatically. This seems to be something of a paradox to me.

Approaches 2 and 3 will only produce identical results if the $XY$ data are standardized within each group, prior to running the regression. This makes the regression analysis somewhat similar to Fisher's test (to my intuition, at least). For unstandardized data, the two regression tests usually agree but diverge more depending on two conditions:

  1. Larger $Y$ variance in one group versus another
  2. Moderate divergence of $XY$ correlation in the two groups

From what I've run, condition 1 seems to have an impact only to the extent that condition 2 is also an issue. When the two tests diverge strongly, one of the two models usually shows heteroscedasticity problems. The fastest way to spot something is wrong is by comparing the regression output to Fisher's test, though. I have added a picture of some data below, where the slopes differ dramatically between groups depending on whether you regress on $X$ or $Y$.

If someone could weigh in with some theoretical insight it would be most appreciated!

Dramatic difference between regression tests

Source Link
Ben M.
  • 205
  • 3
  • 10

Now that I've run a couple of tests in R, I think I understand the nature of the problem a little better, so I'll make an attempt at answering my own question. To rephrase the original problem, basically the test of interest boils down to comparing an association between two independent groups. The way I see it there are three approaches to this comparison:

  1. Fisher's test for comparing independent correlations
  2. Regressing $Y$ on $X$ with a grouping variable $G$ and testing the interaction (as above)
  3. Regressing $X$ on $Y$ with a grouping variable $G$ and testing the interaction (as above)

Conceptually, all three tests ask whether some $XY$ association is significantly different in one group versus another. Having run many examples, however, I found that these three tests do not always agree, and sometimes even diverge dramatically. This seems to be something of a paradox to me.

Approaches 2 and 3 will only produce identical results if the $XY$ data are standardized within each group, prior to running the regression. This makes the regression analysis somewhat similar to Fisher's test (to my intuition, at least). For unstandardized data, the two regression tests usually agree but diverge more depending on two conditions:

  1. Larger $Y$ variance in one group versus another
  2. Moderate divergence of $XY$ correlation in the two groups

From what I've run, condition 1 seems to have an impact only to the extent that condition 2 is also an issue. When the two tests diverge strongly, one of the two models usually shows heteroscedasticity problems. The fastest way to spot something is wrong is by comparing the regression output to Fisher's test, though.

If someone could weigh in with some theoretical insight it would be most appreciated!