Two correlated DVs in a multiple regression

Question

I have a multiple regression with an interaction term and I would like to run it on two separate DVs. The two DVs are quite highly intercorrelated (but conceptually different). How can I control for that intercorrelation?

Answer 1

The original question suggests to me the "seemingly unrelated regression" approach (SUR). As pointed out, the two equations could be estimated separately by ordinary least squares. However, the advantage of this approach over estimating 2 separate equations is that SUR produces more efficient estimates. Moreover, with SUR you have a nice framework to test cross-equation hypotheses.

In R there is a package called systemfit. This package has a nice vignette explaining the different approaches. That could be a good starting point, even if you do not plan to use R. Otherwise, have a look at an econometrics textbook. Most of them cover the estimation of systems of equations.

Be careful. If you obtain results that are not in line with your expectations, there may be many explanations. I do not see why the lack to "control for this correlation" must be the blemish. There can be lots of other reasons, misspecification, measurement error, wrong expectations, ...

Answer 2

As others I do not really understand what you mean by "control for intercorrelation" But from your comment I give it a try with a variant which I can imagine to be possibly meaningful. To show this in principle I begin with a random covariance matrix in the sense of three iv's $x_1,x_2,x_3$ and two dv's $y_1,y_2$

$ \qquad \small \text{ cov =} \begin{array} {r|rrr|rr} &x1&x2&x3&y1&y2\\ \hline x1&1.0&0.6640&0.6160&0.7006&0.6237\\ x2&0.6640&1.0&0.8133&0.9745&0.6906\\ x3&0.6160&0.8133&1.0&0.7633&0.7318\\ \hline y1&0.7006&0.9745&0.7633&1.0&0.7918\\ y2&0.6237&0.6906&0.7318&0.7918&1.0\\ \hline \end{array} $

From this we get a triangular cholesky-decomposition which shows the "factorloadings" in the sense of principal components analysis.

$ \qquad \small \text{ L =} \begin{array} {r|rrr|rr} &f1&f2&f3&f4&f5\\ \hline x1&1.0&.&.&.&.\\ x2&0.6640&0.7477&.&.&.\\ x3&0.6160&0.5407&0.5729&.&.\\ \hline y1&0.7006&0.6811&-0.0639&0.2028&.\\ y2&0.6237&0.3697&0.2579&0.5891&0.2465\\ \hline \end{array} $

The regression-parameters $\beta$ can be obtained by postmultiplication with the inverse of the correlated x-loadings, the inverse of the 3x3-top-left partial loadingsmatrix:

$ \qquad \small L_{xx}^{-1} = \begin{array} {rrr} 1.0& .&.\\ -0.8881&1.3374&.\\ -0.2371&-1.2624&1.7456 \end{array} $

and then the $\beta$ occur by postmultiplication into the first three columns and can be taken from the rows of the dv's. The squares of the values in the 4'th and 5'th columns are the unexplained variances of the dv's, where 0.2028*0.5891 is the partial correlation of the dv's which is independent of the iv's and 0.2465^2 is the variance of y2 which is uncorrelated with all.

$ \qquad \small \beta = \begin{array} {r|rrr|rr} &x1&x2&x3&f4&f5\\ \hline x1&1.0&.&.&.&.\\ x2&.&1.0&.&.&.\\ x3&.&.&1.0&.&.\\ \hline y1&0.1109&0.9916&-0.1115&0.2028&.\\ y2&0.2342&0.1690&0.4501&0.5891&0.2465\\ \hline \end{array} $

Now we attack the problem of "controlling for intercorrelation".

a) If that means, we want the regression after y1 is partialled out, then we can use this loadings-matrix in a different rotational position. I rotate it such that the variance of the y1 is collected in the first column and the regression can be done on the remaining variance. Here is the rotated loadingsmatrix:

$ \qquad \small L_2 = \begin{array} {r|r|rrr|r} &f1&f2&f3&f4&f5\\ \hline x1&0.7006&0.7135&.&.&.\\ x2&0.9745&-0.0263&0.2228&.&.\\ x3&0.7633&0.1138&0.3254&0.5464&.\\ \hline y1&1.0000&.&.&.&.\\ \hline y2&0.7918&0.0966&-0.3523&0.4230&0.2465\\ \hline \end{array} $

We see, that the columns 2 to 5 are free of variance of the dv1, and we can do the regression by postmultiplication with the inverse of the partial matrix $L_{1..3,2..4} $ We get the following:

$ \qquad \small \beta_2 = \begin{array} {r|r|rrr|r} &y1&x1._{y1}&x2._{y1}&x3._{y1}&f5\\ \hline x1&0.7006&1.0&.&.&.\\ x2&0.9745&.&1.0&.&.\\ x3&0.7633&.&.&1.0&.\\ \hline y1&1.0&.&.&.&.\\ \hline y2&0.7918&-0.0880&-2.7114&0.7741&0.2465\\ \hline \end{array} $

and in the last row we find $\beta$-coefficients, which may be understood as the regression where the dv1 is held constant. As well as we partialled out the dv1 we could else partial out the principal component of y1 and y2 (which is then possibly equal to "canonical correlation")

b) Here is such an example. I rotated for the principal components of the dv's only. That variance is now represented in the first two columns.

$ \qquad \small L_3= \begin{array} {r|rr|rrr} &p1&p2&f3&f4&f5\\ \hline x1&0.6996&0.1193&-0.4539&-0.2487&-0.4780\\ x2&0.8796&0.4400&-0.1000&0.1487&0.0255\\ x3&0.7898&0.0488&-0.5012&0.2810&0.2091\\ \hline y1&0.9465&0.3227&.&.&.\\ y2&0.9465&-0.3227&.&.&.\\ \hline \end{array} $

To see the variance of the second principal component of the dv's for the regression on x I rotate the second to 5'th column to the triangular form (this is not really needed for the proceeding)

$ \qquad \small L_4 = \begin{array} {r|r|rrr|r} &p1&f2&f3&f4&f5\\ \hline x1&0.6996&0.7146&.&.&.\\ x2&0.8796&0.0681&0.4708&.&.\\ x3&0.7898&0.0889&0.2392&0.5578&.\\ \hline y1&0.9465&0.0539&0.2937&-0.1063&0.0603\\ y2&0.9465&-0.0539&-0.2937&0.1063&-0.0603\\ \hline \end{array} $

Now we do the regression on the variance of the second principal component only. We get by postmultiplication of the inverse of the partialled x-variance the following matrix of beta-values:

$ \qquad \small \beta_4 = \begin{array} {r|r|rrr|r} &p1&x1._{p1}&x2.{p1}&x3.{p1}&f5\\ \hline x1&0.6996&1.0&.&.&.\\ x2&0.8796&.&1.0&.&.\\ x3&0.7898&.&.&1.0&.\\ \hline y1&0.9465&0.0303&0.7206&-0.1906&0.0603\\ y2&0.9465&-0.0303&-0.7206&0.1906&-0.0603\\ \hline \end{array} $

Now we have regression weights for the second principal component of y1 and y2 on the partialled iv's x1,x2 and x3, and might say, that the "intercorrelation of dv1 and dv2" was controlled - in the sense that the most common variance/their first principal component was held constant.

I don't know whether this line of attack goes in the direction where you want to go to, but perhaps it comes near.