# Regression with correlated explanatory variables

I have variables of the following kind (coded in R):

set.seed(2)
dependent.variable = rnorm(12)
exp1 = c(1,3,4,8,3,4,1,5,6,6,7,9)
exp2 = c(1,3,6,2,1,1,3,4,6,4,1,1)
exp3 = exp1*exp2


my three explanatory variables are related by an equation of the kind: $exp1 = \frac{exp3}{exp2}$. For my real variables the relation might be slightly more complex and I haven't figured out the equation yet.

I want to know if any of these three variables influence dependent.variable. Does it make sense?

Can I simply run:

lm(dependent.variable~exp1*exp2*exp3)


or

lm(dependent.variable~poly(exp1,2,raw=T)*poly(exp2,2,raw=T)*poly(exp3,2,raw=T))


without having to care about the fact that the explanatory variables are not independent? If not, what should I do?

Thanks a lot!

• Question: I am not familiar with the R-code. Can you write the regression equation specification in mathematical symbols, so that I can understand whether you are running a linear or non-linear regression? Aug 24, 2013 at 15:25
• @AlecosPapadopoulos The first command line is a linear model, the second one is a quadratic model. In each case, estimations for all interactions are considered. Aug 25, 2013 at 11:14
• What do you mean exactly when you say the explanatory variables are related by an equation but you don't know what it is? If knowing values of any two allows you to derive the third, then any multicollinearity is merely structural. Aug 25, 2013 at 11:35
• I have performed simulations and measured three variables which are correlated. But I haven't created the third variable thanks to the two others as I did in example in my post? Does it answer your comment? I think, for the purpose of this question, it doesn't matter what is the relation between my three explanatory variables. Let's assume the relation is of the kind $exp3=exp1*EXP2$. In such a case, how should I take into account their correlation? Hope this makes sense! Aug 25, 2013 at 11:44
• (1) For your example, follow @AlecosPapadopoulos's suggestion & write out your models. You'll see why they're silly - you're sometimes putting two regression coefficients to what is really the same term. (2) If any one explanatory variable is not exactly determined by the other two, but they're highly correlated, then what you have is multicollinearity - check the tag & you'll see plenty of advice on how to measure its effects on your model & how to deal with it. Aug 25, 2013 at 12:56

In the example you give it would make no sense to talk about only two of the explanatory variables ($x_1$, $x_2$) influencing a dependent variable ($y$), as the third ($x_3$) is derived from them. For e.g. the linear model with interactions, the fitted value of $y$ is given by

$$\hat{y} = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \beta_3 x_3 + \beta_{12} x_1 x_2 + \beta_{13} x_1 x_3 + \beta_{23} x_2 x_3$$

where $\beta$ are the coefficients you want to estimate. Substituting e.g. $x_3 = x_1 x_2$ gives

$$\hat{y} = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \beta_3 x_1 x_2 + \beta_{12} x_1 x_2 + \beta_{13} x_1^2 x_2 + \beta_{23} x_1 x_2^2$$

so $\beta_3$ & $\beta_{12}$ are coefficients for the same term, & you can't separately estimate them. You could fit

$$\hat{y} = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \beta_{3}^* x_3 + \beta_{13} x_1 x_3 + \beta_{23} x_2 x_3$$

if this model is of interest; & the multicollinearity is merely structural. (Note though that it forces the relation of the dependent variable to both $x_1$ & $x_2$ to be linear when $x_3=0$.)

In general if an explanatory variable is a function of the other explanatory variables it's simpler to omit it—you can always rewrite the fitted model to put it back in. Of course there's no guarantee that the dependent variable is well fitted by a simple additive model.

But when you say you measured three correlated variables it makes me doubt that the correlation is perfect, as in your example. If it's not then there are plenty of questions on this site about how to assess the effects of multicollinearity & how to deal with it, & the 'multicollinearity' tag will help you find them.

If you have some knowledge of a hypothetical relation (e.g., from the literature), then you might be interested in looking at non-linear as well as linear regression models. Standing on the shoulders of those who have worked on this stuff before you can give you some great vision.

If you are content to limit yourself to additive linear regression models, then I suggest you start with two independent variables and their interaction:

lm(dependent.variable ~ exp1 + exp2 + exp1:exp2)


If there are signs of non-linearity in this relation, you may wish to explore the nature of that non-linearity with a generalized additive model.

library(mgcv)
gam(dependent.variable ~ s(exp1) + s(exp2) + s(exp3))


Note that this last line of code will give you an error in R with your example data set because there are so few observations. If you get the same error with your full data set, use the argument k= in the s() function to limit the degrees of freedom used in each smooth.