# Two correlated DVs in a multiple regression

I have a multiple regression with an interaction term and I would like to run it on two separate DVs. The two DVs are quite highly intercorrelated (but conceptually different). How can I control for that intercorrelation?

-
I suggest you think in very precise terms about what you mean by "control" in this context. Within one multiple regression, control would mean to see how X1 relates to Y while partialling out or holding constant X2. What does control mean with respect to your situation involving a second Y? – rolando2 Aug 7 '11 at 14:13
Under the standard linear regression assumptions, fitting a "joint" (multivariate response) linear regression is the same as fitting single-response linear regressions to each of the response variables separately. This is a standard result that can be found in most any linear-regression text, e.g., G. A. F. Seber and A. Lee (2003), Linear Regression Analysis, Wiley, 2nd ed. – cardinal Aug 7 '11 at 14:57
@Cardinal There may be an issue here with protecting significance: there will be multiple tests which, due to the correlation of the DVs, could be construed to be strongly dependent. – whuber Aug 7 '11 at 15:14
By control I mean how to account for the common variance shared by the DVs. I'm getting a similar pattern of results for both DVs, atlhough theoretically it should be different. I guess my follow-up question would be weather it could be a solution to control for the other DV, while running each of the regression analyses? – ale Aug 7 '11 at 16:03

The original question suggests to me the "seemingly unrelated regression" approach (SUR). As pointed out, the two equations could be estimated separately by ordinary least squares. However, the advantage of this approach over estimating 2 separate equations is that SUR produces more efficient estimates. Moreover, with SUR you have a nice framework to test cross-equation hypotheses.

In R there is a package called systemfit. This package has a nice vignette explaining the different approaches. That could be a good starting point, even if you do not plan to use R. Otherwise, have a look at an econometrics textbook. Most of them cover the estimation of systems of equations.

Be careful. If you obtain results that are not in line with your expectations, there may be many explanations. I do not see why the lack to "control for this correlation" must be the blemish. There can be lots of other reasons, misspecification, measurement error, wrong expectations, ...

-
 Thanks! I will look into that. Of course I do realize reasons might be many (including wrong theory) but this was the one I did not know how to deal with (or if I can deal with). – ale Aug 7 '11 at 21:46 Though I could very well be wrong, it seems like the OP is considering regression on two responses using the same set of regressors. I fail to see how SUR would be different than OLS in this case or how it would be more efficient. Perhaps you can expand on this. The separability of the regressions does break down if the covariance of the errors varies by observation. – cardinal Aug 7 '11 at 21:48

As others I do not really understand what you mean by "control for intercorrelation" But from your comment I give it a try with a variant which I can imagine to be possibly meaningful. To show this in principle I begin with a random covariance matrix in the sense of three iv's $x_1,x_2,x_3$ and two dv's $y_1,y_2$

$\qquad \small \text{ cov =} \begin{array} {r|rrr|rr} &x1&x2&x3&y1&y2\\ \hline x1&1.0&0.6640&0.6160&0.7006&0.6237\\ x2&0.6640&1.0&0.8133&0.9745&0.6906\\ x3&0.6160&0.8133&1.0&0.7633&0.7318\\ \hline y1&0.7006&0.9745&0.7633&1.0&0.7918\\ y2&0.6237&0.6906&0.7318&0.7918&1.0\\ \hline \end{array}$

From this we get a triangular cholesky-decomposition which shows the "factorloadings" in the sense of principal components analysis.

$\qquad \small \text{ L =} \begin{array} {r|rrr|rr} &f1&f2&f3&f4&f5\\ \hline x1&1.0&.&.&.&.\\ x2&0.6640&0.7477&.&.&.\\ x3&0.6160&0.5407&0.5729&.&.\\ \hline y1&0.7006&0.6811&-0.0639&0.2028&.\\ y2&0.6237&0.3697&0.2579&0.5891&0.2465\\ \hline \end{array}$

The regression-parameters $\beta$ can be obtained by postmultiplication with the inverse of the correlated x-loadings, the inverse of the 3x3-top-left partial loadingsmatrix:

$\qquad \small L_{xx}^{-1} = \begin{array} {rrr} 1.0& .&.\\ -0.8881&1.3374&.\\ -0.2371&-1.2624&1.7456 \end{array}$

and then the $\beta$ occur by postmultiplication into the first three columns and can be taken from the rows of the dv's. The squares of the values in the 4'th and 5'th columns are the unexplained variances of the dv's, where 0.2028*0.5891 is the partial correlation of the dv's which is independent of the iv's and 0.2465^2 is the variance of y2 which is uncorrelated with all.

$\qquad \small \beta = \begin{array} {r|rrr|rr} &x1&x2&x3&f4&f5\\ \hline x1&1.0&.&.&.&.\\ x2&.&1.0&.&.&.\\ x3&.&.&1.0&.&.\\ \hline y1&0.1109&0.9916&-0.1115&0.2028&.\\ y2&0.2342&0.1690&0.4501&0.5891&0.2465\\ \hline \end{array}$

Now we attack the problem of "controlling for intercorrelation".

a) If that means, we want the regression after y1 is partialled out, then we can use this loadings-matrix in a different rotational position. I rotate it such that the variance of the y1 is collected in the first column and the regression can be done on the remaining variance. Here is the rotated loadingsmatrix:

$\qquad \small L_2 = \begin{array} {r|r|rrr|r} &f1&f2&f3&f4&f5\\ \hline x1&0.7006&0.7135&.&.&.\\ x2&0.9745&-0.0263&0.2228&.&.\\ x3&0.7633&0.1138&0.3254&0.5464&.\\ \hline y1&1.0000&.&.&.&.\\ \hline y2&0.7918&0.0966&-0.3523&0.4230&0.2465\\ \hline \end{array}$

We see, that the columns 2 to 5 are free of variance of the dv1, and we can do the regression by postmultiplication with the inverse of the partial matrix $L_{1..3,2..4}$ We get the following:

$\qquad \small \beta_2 = \begin{array} {r|r|rrr|r} &y1&x1._{y1}&x2._{y1}&x3._{y1}&f5\\ \hline x1&0.7006&1.0&.&.&.\\ x2&0.9745&.&1.0&.&.\\ x3&0.7633&.&.&1.0&.\\ \hline y1&1.0&.&.&.&.\\ \hline y2&0.7918&-0.0880&-2.7114&0.7741&0.2465\\ \hline \end{array}$

and in the last row we find $\beta$-coefficients, which may be understood as the regression where the dv1 is held constant. As well as we partialled out the dv1 we could else partial out the principal component of y1 and y2 (which is then possibly equal to "canonical correlation")

b) Here is such an example. I rotated for the principal components of the dv's only. That variance is now represented in the first two columns.

$\qquad \small L_3= \begin{array} {r|rr|rrr} &p1&p2&f3&f4&f5\\ \hline x1&0.6996&0.1193&-0.4539&-0.2487&-0.4780\\ x2&0.8796&0.4400&-0.1000&0.1487&0.0255\\ x3&0.7898&0.0488&-0.5012&0.2810&0.2091\\ \hline y1&0.9465&0.3227&.&.&.\\ y2&0.9465&-0.3227&.&.&.\\ \hline \end{array}$

To see the variance of the second principal component of the dv's for the regression on x I rotate the second to 5'th column to the triangular form (this is not really needed for the proceeding)

$\qquad \small L_4 = \begin{array} {r|r|rrr|r} &p1&f2&f3&f4&f5\\ \hline x1&0.6996&0.7146&.&.&.\\ x2&0.8796&0.0681&0.4708&.&.\\ x3&0.7898&0.0889&0.2392&0.5578&.\\ \hline y1&0.9465&0.0539&0.2937&-0.1063&0.0603\\ y2&0.9465&-0.0539&-0.2937&0.1063&-0.0603\\ \hline \end{array}$

Now we do the regression on the variance of the second principal component only. We get by postmultiplication of the inverse of the partialled x-variance the following matrix of beta-values:

$\qquad \small \beta_4 = \begin{array} {r|r|rrr|r} &p1&x1._{p1}&x2.{p1}&x3.{p1}&f5\\ \hline x1&0.6996&1.0&.&.&.\\ x2&0.8796&.&1.0&.&.\\ x3&0.7898&.&.&1.0&.\\ \hline y1&0.9465&0.0303&0.7206&-0.1906&0.0603\\ y2&0.9465&-0.0303&-0.7206&0.1906&-0.0603\\ \hline \end{array}$

Now we have regression weights for the second principal component of y1 and y2 on the partialled iv's x1,x2 and x3, and might say, that the "intercorrelation of dv1 and dv2" was controlled - in the sense that the most common variance/their first principal component was held constant.

I don't know whether this line of attack goes in the direction where you want to go to, but perhaps it comes near.

-