Interaction and Mediation in linear regression

Question

I am struggling with some analysis I'm doing and would appreciate some insights:

I'm measuring people's dependency on "Environmental attitude" (DV) on the number of hours of sports activities they are doing "out in nature" (IV 1) and inside a building (IV 2) [the data is from a survey N~500). When I run a linear regression with these two I get that IV1 (nature hours) has a positive significant effect on the DV, and IV2 does not.

Then I added another IV: the ratio between time the activities in the following form:

$$ \text{IV3} = \frac{\text{time outside}-\text{time inside}}{\text{time outside} + \text{time inside}} + 1 $$

so it is a variable that ranges from 0 to 2, where 0 indicates only inside activities and 2 indicates only outside activities (and 1 is an equal amount of time).

IV3 is highly correlated to IV1 and IV2.

When I run a regression with the 3 of them, the R-square gets better (more of the variance is explained) but to my surprise IV2 (hours inside, which wasn't significant before) becomes significant as well as IV3 (the ratio) while IV1 becomes not-significant.

Why do you think that is happening? How would you approach this analysis?

Thanks!

Answer 1

1. Don't ever be surprised that $r^2$ increases when you add a new variable to the regression, even one that's highly correlated with other predictors. If the new variable is a linear function of the old independent variables (i.e. if you had just included IV1 - IV2 as a new variable), then your software would probably give you an error message. Otherwise, the new independent variable will "soak up" some variance and increase $r^2$. If you haven't seen this before, generate some completely random numbers, add them as a new independent variable to your regression, and watch $r^2$ increase!

2. Hard to say exactly why the "significance" changes when you add IV3 to the model. One possibility is that you can almost compute IV1 as a linear combination of IV2 and IV3, and that approximation of IV1 does a better job predicting than IV1. But this isn't really what you should be focusing on. You need to ask yourself, "What is my scientific hypothesis, and how should I go about fitting a model that can test this hypothesis?"

For instance, if you simply think that the DV is positively associated with IV1 and negatively with IV2, just include those 2 in the model. Your initial result doesn't surprise me: people who exercise outdoors more probably do love the environment more, and people who exercise indoors more probably don't dislike the environment (they're still active), they just like to lift or play basketball in the gym or what have you. If you suspect there is something more complicated at play, you might fit an "interaction term."

Funky things happen when you start adding highly correlated variables to models. Rather than testing a bunch of models and seeing what's significant and what isn't (this is bad scientific practice), build a model that stems from the scientific background and diagnose it carefully before trying something else.

Answer 2

First, in addition to Sheldon's points, looking at significant vs. not is not good practice. See Andrew Gelman's paper "The Difference Between `Significant' and 'Not Significant' is not, itself, Statistically Significant."

Second, you are going to have collinearity. This makes computation of parameter estimates, their standard errors, and their p values problematic. There are various ways to deal with it, but, here, I think the best might be ridge regression. This allows some bias in the results in exchange for greatly decreasing the standard errors of the parameter estimates.