3
$\begingroup$

Following the advice here regarding statistical significance versus substantive meaning of a model, I did a scatterplot with 'fit line at subgroups'.

My graph clearly shows that there is an interaction between X1 and X2 on Y. However, the F test of the X1*X2 interaction is insignificant. I am using SPSS GLM. X1 is significant but X2 is insignificant and the product term (X1*X2) is insignificant as well.

Question: How do I reconcile these contradictory outcomes? (i.e. see an interaction in the graph but not seeing a significant F test result in GLM)

I understand the importance of substantive meaning. However, sooner or later, someone may ask me why I am graphing the X1 and X2 on Y when the product term is insignificant.

(In the Parameter Estimates table in the SPSS GLM outcome, the t test for each level of X1 with X2 (e.g. maleX1 and femaleX1) is signficant.)

Please see graph below:

graph

Improve this question
$\endgroup$
6
  • $\begingroup$ Hint: think about what's required for an anova effect to show up as statistically significant. The ratio of what to what must be large? And now, have you examined that ratio, or just looked at means? $\endgroup$
    – rolando2
    Dec 17, 2011 at 13:34
  • 1
    $\begingroup$ Reproducing your graph here would help, Adhesh. My suspicion is that the interaction appears insignificant because it might not be of the form X1*X2: it could be more complicated than that. $\endgroup$
    – whuber ♦
    Dec 17, 2011 at 14:12
  • $\begingroup$ Thanks whuber! I just learnt how to import a graph in my question above. Wish things were as simple as this! $\endgroup$
    – Adhesh Josh
    Dec 17, 2011 at 23:07
  • 3
    $\begingroup$ It was good that you displayed the graph. "My graph clearly shows that there is an interaction...." If that's clear evidence for an interaction, think about how similar the slopes would have to be for it to be "unclear." Let alone for clear evidence of no interaction. $\endgroup$
    – rolando2
    Dec 18, 2011 at 0:20
  • 1
    $\begingroup$ Thanks for the scatterplot, Adhesh. The difference in slopes sure looks insignificant to me--the plot certainly does not contradict the SPSS results--but it's hard to tell because it's unclear how much data are involved and I have no sense of how important a difference in Y values of 0.15 or so might be. You can get more insight by running a Lowess smooth of Y against X1 (and vice-versa, if that's relevant) and asking whether the sizes of any variations seen there are important in your investigation. $\endgroup$
    – whuber ♦
    Dec 19, 2011 at 15:08

2 Answers 2

Reset to default
6
$\begingroup$

Well... the slopes on the graph you show are the sames. Well, not exactly the same, but too close to be sanely considered as really different (taking into account the small number of data points).

The following R code generates a plot roughly similar to your plot. The slopes of the two regression lines are estimates of the same "true value" of 0.8. They are slightly different of 0.8, it’s because they are just estimates of an unknown true value.

X1 <- rnorm(100, 2.5)
X2 <- rnorm(100, 2.5)
Y1 <- 0.8*X1 + rnorm(100)
Y2 <- 0.8*X2 + rnorm(100)
plot(X1, Y1, pch=15)
points(X2, Y2, pch=19)
abline(lm(Y1~X1))
abline(lm(Y2~X2),lty=2)

similar plot

I hope that seeing this phenomenon on simulated data (where you can see the "true value") will help you to understand.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks! I think I am slowly getting it. i.e. the effect of X1 on Y is the same for gender i.e. there is no difference between male amd female - and this is the reason why the product term, X1*Gender(X2), is insignificant. Is this correct? I am using SPSS. $\endgroup$
    – Adhesh Josh
    Dec 18, 2011 at 13:46
  • $\begingroup$ I just want to show you that when you see two different slopes, that doesn’t prove that the "true slopes" are different. I don’t know SPSS but I think you can easily adapt the code. X1 and X2 are vectors of 100 independent normal random variables with mean 2.5 and standard deviation 1. Then Y1 = 0.8 X1 + ε where ε is an error term which is a standard normal random variable ; same thing for Y2. Then there’s a plot similar to yours, with the two regression lines computed for Y1 ~ X1 and Y2 ~ X2. You see two distinct lines, with distinct slopes, even if the "true slopes" are the same (0.8). $\endgroup$
    – Elvis
    Dec 18, 2011 at 17:24
  • $\begingroup$ @AdheshJosh Thanks for accepting! When I answered I hadn’t enough reputation to insert the graph, now this is done. $\endgroup$
    – Elvis
    Jan 16, 2012 at 10:23
  • $\begingroup$ Thanks. This makes a lot of sense. Since asking this question, I have found that the difference in slopes can be attributed to sampling error. In other words, it is identical regression for male and female. $\endgroup$
    – Adhesh Josh
    Jan 16, 2012 at 11:02
3
$\begingroup$

Statistical significance is partly a function of sample size. Large effects are interesting, whether they are significant or not. Even Fisher said that you should test everything with the same p-value (I read a quote from him somewhere along the lines of "no sane researcher would use the same critical value all the time" but I can't find it right now).

You could read Ziliak and McCloskey The Cult of Statistical Significance or, for a less polemical view, Abelson's Statistics as Principled Argument. I reviewed the latter on my blog; but, briefly, Abelson argues that we should evaluate statistical arguments on the following criteria

  1. Magnitude How big is the effect?
  2. Articulation How precisely stated is it?
  3. Generality How widely does it apply?
  4. Interesting How interesting is it?
  5. Credibility How believable is it?

Notice that "significance" is not listed.

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ Isn't the Fisher reference missing an emphatic "not"? $\endgroup$
    – whuber ♦
    Dec 17, 2011 at 14:20
  • 3
    $\begingroup$ "(...) no scientific worker has a fixed level of significance at which from year to year, and in all circumstances, he rejects hypotheses; he rather gives his mind to each particular case in the light of his evidence and his ideas." (Fisher, 1956, p. 41) See also Why P=0.05? $\endgroup$
    – chl
    Dec 17, 2011 at 16:58
  • $\begingroup$ Thanks Peter. This makes a lot of sense but with the obession on statistically significant results, it is a bit tricky. $\endgroup$
    – Adhesh Josh
    Dec 18, 2011 at 7:56
  • $\begingroup$ Who is obsessing about significance, Adhesh? I see little of that anywhere on this site, anyway. $\endgroup$
    – whuber ♦
    Dec 19, 2011 at 15:05
  • $\begingroup$ @whuber. That was just my general observation and totally unrelated to this site. The question now is should I or should I not describe the trend in above graph as an interaction. (My research context implies that there is an interaction.) $\endgroup$
    – Adhesh Josh
    Dec 21, 2011 at 0:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.