Regression with correlated explanatory variables

Question

I have variables of the following kind (coded in R):

set.seed(2)
dependent.variable = rnorm(12)
exp1 = c(1,3,4,8,3,4,1,5,6,6,7,9)
exp2 = c(1,3,6,2,1,1,3,4,6,4,1,1)
exp3 = exp1*exp2

my three explanatory variables are related by an equation of the kind: $exp1 = \frac{exp3}{exp2}$. For my real variables the relation might be slightly more complex and I haven't figured out the equation yet.

I want to know if any of these three variables influence dependent.variable. Does it make sense?

Can I simply run:

lm(dependent.variable~exp1*exp2*exp3)

or

lm(dependent.variable~poly(exp1,2,raw=T)*poly(exp2,2,raw=T)*poly(exp3,2,raw=T))

without having to care about the fact that the explanatory variables are not independent? If not, what should I do?

Thanks a lot!

Answer 1

In the example you give it would make no sense to talk about only two of the explanatory variables ($x_1$, $x_2$) influencing a dependent variable ($y$), as the third ($x_3$) is derived from them. For e.g. the linear model with interactions, the fitted value of $y$ is given by

$$\hat{y} = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \beta_3 x_3 + \beta_{12} x_1 x_2 + \beta_{13} x_1 x_3 + \beta_{23} x_2 x_3 $$

where $\beta$ are the coefficients you want to estimate. Substituting e.g. $x_3 = x_1 x_2$ gives

$$\hat{y} = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \beta_3 x_1 x_2 + \beta_{12} x_1 x_2 + \beta_{13} x_1^2 x_2 + \beta_{23} x_1 x_2^2 $$

so $\beta_3$ & $\beta_{12}$ are coefficients for the same term, & you can't separately estimate them. You could fit

$$\hat{y} = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \beta_{3}^* x_3 + \beta_{13} x_1 x_3 + \beta_{23} x_2 x_3 $$

if this model is of interest; & the multicollinearity is merely structural. (Note though that it forces the relation of the dependent variable to both $x_1$ & $x_2$ to be linear when $x_3=0$.)

In general if an explanatory variable is a function of the other explanatory variables it's simpler to omit it—you can always rewrite the fitted model to put it back in. Of course there's no guarantee that the dependent variable is well fitted by a simple additive model.

But when you say you measured three correlated variables it makes me doubt that the correlation is perfect, as in your example. If it's not then there are plenty of questions on this site about how to assess the effects of multicollinearity & how to deal with it, & the 'multicollinearity' tag will help you find them.

Answer 2

If you have some knowledge of a hypothetical relation (e.g., from the literature), then you might be interested in looking at non-linear as well as linear regression models. Standing on the shoulders of those who have worked on this stuff before you can give you some great vision.

If you are content to limit yourself to additive linear regression models, then I suggest you start with two independent variables and their interaction:

lm(dependent.variable ~ exp1 + exp2 + exp1:exp2)

If there are signs of non-linearity in this relation, you may wish to explore the nature of that non-linearity with a generalized additive model.

library(mgcv)
gam(dependent.variable ~ s(exp1) + s(exp2) + s(exp3))

Note that this last line of code will give you an error in R with your example data set because there are so few observations. If you get the same error with your full data set, use the argument k= in the s() function to limit the degrees of freedom used in each smooth.