How to extract dependence on a single variable when independent variables are correlated?

Question

I have a dataset in which I have measured a dependent variable (let's call it $Y$) along with several independent variables $(X_1, X_2, X_3)$. The independent variables are correlated with one another to some extent. I would like to understand how $Y$ varies with $X_2$ when $X_1$ and $X_3$ are held constant. What approach will allow me to extract this relationship given the correlation between the independent variables?

I have looked into principal component analysis, but that casts the data in terms of linear combinations that include $X_2$, thus not separating the $X_2$ dependence.

A example dataset (csv format).

Answer 1

Aksakal's answer is correct. By controlling for all variables in a regression, you "keep them constant" and you are able to identify the partial correlation between your regressor of interest. Let me give you an example to make this clearer.

First, let us create some correlated $X$s.

 ex <- rnorm(1000)
 x1 <- 5*ex + rnorm(1000)
 x2 <- -3*ex + rnorm(1000)
 x3 <- 4*ex + rnorm(1000)

Now, since all these variables are generated by some underlying variable $ex$, they are clearly correlated. You can check this using cor(x1,x2), for instance.

Now, let us generate the dependent variable with known parameters.

 y <- 1*x1 + 2*x2 + 3*x3 + rnorm(1000)

Here we know that $\beta_1=1, \beta_2=2, \beta_3=3$. I have picked them arbitrarily. Let us now see if Aksakal's approach can uncover these parameters:

 lm(y ~ x1+x2+x3)

If it works, the estimated parameters should be close to the ones we have picked. Here the result:

 Call:
 lm(formula = y ~ x1 + x2 + x3)

 Coefficients:
 (Intercept)           x1           x2           x3  
    -0.01224      0.99805      1.99746      2.99670  

As you can see, all parameters have been uncovered.

Having said that, there are many caveats involved here as well. Most importantly, you should not interpret these coefficients in a causal way. Depending on your actual situation, it might help if you explain a bit more what you are trying to estimate so that people can evaluate whether this method is appropriate (or whether answering your research question is feasible at all). For instance, why do you think your independent variables are correlated? Is it that $X_1$ might have an effect on $X_2$ and this has an effect on $y$? If this is the setup you have in mind, then depending on your field, you may want to look into mediator/moderator analysis or into quasi-experimental methods. Hence you see you might benefit from elaborating a bit more on your situation.

Answer 2

Regress Y on Xs, beta of X2 will be what you are looking for.

UPDATE:

I'll add to my answer based on the discussion after my original post.

Consider $y=f(x_1,x_2,x_3)$, an arbitrary smooth function. It seems that you are looking for the sensitivity of $y$ to $x_2$. This is captured by the partial derivative $\partial y/\partial x_2$. To see this it helps to look at Taylor expansion: $y(x+\Delta x)=y(x)+\partial y/\partial x_1 \Delta x_1+\partial y/\partial x_2 \Delta x_2+\partial y/\partial x_3 \Delta x_3+\partial^2 y/\partial x_1^2 (\Delta x_1)^2+\partial^2 y/(\partial x_1 \partial x_2) \Delta x_1 \Delta x_2+...$.

Note, how the interaction terms are of the second order in $\Delta$'s. So, if you're interested in the first order effects, then you are looking for $\partial f/\partial x_2$, i.e. the $\beta_{X_2}$ in your regression. Also note, that this does not preculde you from adding interaction terms in your regression such as $X_2*X_3$ or $X_1*X_2*X_3$. These are fine, but you don't need their coefficients to answer your question. When you add interaction terms, of course, your $\beta_{X_2}$ will change, but its interpretation won't.

Answer 3

Primary to your concern should be whether the model of $Y$ given all $X$ is correct. If it is correct, the $\beta$ of $X_2$ is the effect coefficient you are looking for. Take into account that there may be non-linear trends on any $X$ with $Y$, $Y$ may not be normal (in which case you need a large sample), and there may be interactions betwen any $X$.

In particular effect hetorogeneity is an issue, which may bias your $\beta$ estimates. You should be able to model it, however, by including interaction terms of $X_2$ with the other $X$ in the model. When there are significnat interactions including them in the model will give you better (i.e., unbiased) estimates of the average effect of $X_2$ on $Y$.

Moreover, if you are in the situation of a case-control or an observational study like a quasi- or natural-experiment, as I take from one of your comments above, $X_2$ is actually dichotomous indicating treatment or control. Then there is a series of other approaches for valid inference about the average treatment effect of $X_2$ on $Y$. For example, you could match treatment and control units indicated by $X_2$ conditional on the other $X$, by means of matching algorithms and propensity scores. If you are actually in the situation of a case-control study or a binary variable $X_2$ the literature on causal inference provides these and other methods.

A --correct-- regression model of the type discussed above will also provide a correct treamtent estimate. However, it may be flawed when its basic assumptions (e.g. linearity, homoscedasticity, effect homogeneity, etc.) are violated.

I have once discussed the use of regression models for estimating average treatment effects from observational studies here