# How to extract dependence on a single variable when independent variables are correlated?

I have a dataset in which I have measured a dependent variable (let's call it $Y$) along with several independent variables $(X_1, X_2, X_3)$. The independent variables are correlated with one another to some extent. I would like to understand how $Y$ varies with $X_2$ when $X_1$ and $X_3$ are held constant. What approach will allow me to extract this relationship given the correlation between the independent variables?

I have looked into principal component analysis, but that casts the data in terms of linear combinations that include $X_2$, thus not separating the $X_2$ dependence.

A example dataset (csv format).

• I also do not have enough reputation to comment, but I have an interest in this question for nonparametric models. So for a particular variable $X_i$, we do not have a $\beta_i$, but rather a series estimate of some function $f_i(X_i)$. For example, the generalize additive model takes the form: $$Y = f(X_1) + f(X_2) + f(X_3) + \epsilon$$ Would the proposed tool be able to control for the dependence structure in a nonparametric model when the $X_i$'s are known be to correlated? Commented Apr 29, 2014 at 13:45
• en.wikipedia.org/wiki/Partial_correlation is one such method Commented May 1, 2014 at 12:45

Aksakal's answer is correct. By controlling for all variables in a regression, you "keep them constant" and you are able to identify the partial correlation between your regressor of interest. Let me give you an example to make this clearer.

First, let us create some correlated $X$s.

 ex <- rnorm(1000)
x1 <- 5*ex + rnorm(1000)
x2 <- -3*ex + rnorm(1000)
x3 <- 4*ex + rnorm(1000)


Now, since all these variables are generated by some underlying variable $ex$, they are clearly correlated. You can check this using cor(x1,x2), for instance.

Now, let us generate the dependent variable with known parameters.

 y <- 1*x1 + 2*x2 + 3*x3 + rnorm(1000)


Here we know that $\beta_1=1, \beta_2=2, \beta_3=3$. I have picked them arbitrarily. Let us now see if Aksakal's approach can uncover these parameters:

 lm(y ~ x1+x2+x3)


If it works, the estimated parameters should be close to the ones we have picked. Here the result:

 Call:
lm(formula = y ~ x1 + x2 + x3)

Coefficients:
(Intercept)           x1           x2           x3
-0.01224      0.99805      1.99746      2.99670


As you can see, all parameters have been uncovered.

Having said that, there are many caveats involved here as well. Most importantly, you should not interpret these coefficients in a causal way. Depending on your actual situation, it might help if you explain a bit more what you are trying to estimate so that people can evaluate whether this method is appropriate (or whether answering your research question is feasible at all). For instance, why do you think your independent variables are correlated? Is it that $X_1$ might have an effect on $X_2$ and this has an effect on $y$? If this is the setup you have in mind, then depending on your field, you may want to look into mediator/moderator analysis or into quasi-experimental methods. Hence you see you might benefit from elaborating a bit more on your situation.

• "For instance, why do you think your independent variables are correlated? Is it that $X_1$ might have an effect on $X_2$ and this has an effect on $y$?" This is an important issue to address. If, in your observed data, $X_1$ and $X_2$ are correlated, you may not be able to disentangle the effects of $X_i$ on $y$ from each other. With just the information in your example, it is possible that $X_2$ has a causal effect on $X_1$ and $y$, and it wouldn't be feasible to vary $X_2$ while holding $X_1$ constant. Commented May 5, 2014 at 9:01
• Exactly! That is why quasi-experimental approaches would be the best to use. If this is not possible and/or there is a very good sense of how $X_1$ might affect the other variables, then a including mediator / moderator-style modelling might be a remedy. Commented May 6, 2014 at 12:56

Regress Y on Xs, beta of X2 will be what you are looking for.

UPDATE:

I'll add to my answer based on the discussion after my original post.

Consider $y=f(x_1,x_2,x_3)$, an arbitrary smooth function. It seems that you are looking for the sensitivity of $y$ to $x_2$. This is captured by the partial derivative $\partial y/\partial x_2$. To see this it helps to look at Taylor expansion: $y(x+\Delta x)=y(x)+\partial y/\partial x_1 \Delta x_1+\partial y/\partial x_2 \Delta x_2+\partial y/\partial x_3 \Delta x_3+\partial^2 y/\partial x_1^2 (\Delta x_1)^2+\partial^2 y/(\partial x_1 \partial x_2) \Delta x_1 \Delta x_2+...$.

Note, how the interaction terms are of the second order in $\Delta$'s. So, if you're interested in the first order effects, then you are looking for $\partial f/\partial x_2$, i.e. the $\beta_{X_2}$ in your regression. Also note, that this does not preculde you from adding interaction terms in your regression such as $X_2*X_3$ or $X_1*X_2*X_3$. These are fine, but you don't need their coefficients to answer your question. When you add interaction terms, of course, your $\beta_{X_2}$ will change, but its interpretation won't.

• I assume you mean X2 and that beta is its coefficient in the multiple regression. This doesn't address OP's question because X2 is correlated with the other independent variables. Commented Apr 24, 2014 at 1:51
• He's controlling for other variables. Thanks for pointing to a typo. Commented Apr 24, 2014 at 1:52
• If you are interested in the effect $X_{2}$ has on $Y$, you probably also care about the statistical significance of the effect. While the coefficient does give you what you want in terms of the effect, you have to keep in mind that if $X_{1}$ $X_{2}$ and $X_{3}$ are correlated, they may individually show up as insignificant but be highly significant as a group. Estimates of correlated $X$s can vary quite a bit because, due to the correlation, it can be hard for the model to attribute any effect to a particular $X$. Commented Apr 24, 2014 at 5:30
• If I understand your comment, it sounds like I am out of luck. Since $Y$ is correlated with the $X$s, and the $X$s with each other, coefficients from a multi-variable regression of $Y$ will include the mixed effects. The reason I'm interested in isolating the effect of $X_2$ is to quantify a physical effect, unfortunately in this case I can't carry out an experiment that leaves the other variables unchanged. Commented Apr 24, 2014 at 15:26
• It depends on how much data you have and on the severity of the correlation. Correlated $X$s effectively make your sample size smaller in the sense that you just need more data to tease out the effect. In other words, if your sample size is large enough, then the beta on X2 is what you want. Otherwise, the beta on X2 might be a very noise estimate of the effect. Commented Apr 25, 2014 at 0:55

Primary to your concern should be whether the model of $Y$ given all $X$ is correct. If it is correct, the $\beta$ of $X_2$ is the effect coefficient you are looking for. Take into account that there may be non-linear trends on any $X$ with $Y$, $Y$ may not be normal (in which case you need a large sample), and there may be interactions betwen any $X$.

In particular effect hetorogeneity is an issue, which may bias your $\beta$ estimates. You should be able to model it, however, by including interaction terms of $X_2$ with the other $X$ in the model. When there are significnat interactions including them in the model will give you better (i.e., unbiased) estimates of the average effect of $X_2$ on $Y$.

Moreover, if you are in the situation of a case-control or an observational study like a quasi- or natural-experiment, as I take from one of your comments above, $X_2$ is actually dichotomous indicating treatment or control. Then there is a series of other approaches for valid inference about the average treatment effect of $X_2$ on $Y$. For example, you could match treatment and control units indicated by $X_2$ conditional on the other $X$, by means of matching algorithms and propensity scores. If you are actually in the situation of a case-control study or a binary variable $X_2$ the literature on causal inference provides these and other methods.

A --correct-- regression model of the type discussed above will also provide a correct treamtent estimate. However, it may be flawed when its basic assumptions (e.g. linearity, homoscedasticity, effect homogeneity, etc.) are violated.

I have once discussed the use of regression models for estimating average treatment effects from observational studies here