# Regression with multiple dependent variables?

Is it possible to have a (multiple) regression equation with two or more dependent variables? Sure, you could run two separate regression equations, one for each DV, but that doesn't seem like it would capture any relationship between the two DVs?

-
As in SUR or 3SLS? – An old man in the sea. May 5 '15 at 19:56

Yes, it is possible. What you're interested is is called "Multivariate Multiple Regression" or just "Multivariate Regression". I don't know what software you are using, but you can do this in R.

Here's a link that provides examples.

http://www.psych.yorku.ca/lab/psy6140/lectures/MultivariateRegression2x2.pdf

-
One might add that fitting the regressions separateley is indeed equivalent to the multivariate formulation with a matrix of dependent variables. In R with package mvtnorm installed (1st: multivariate model, 2nd: separate univariate models): library(mvtnorm); X <- rmvnorm(100, c(1, 2), matrix(c(4, 2, 2, 3), ncol=2)); Y <- X %*% matrix(1:4, ncol=2) + rmvnorm(100, c(0, 0), diag(c(20, 30))); lm(Y ~ X[ , 1] + X[ , 2]); lm(Y[ , 1] ~ X[ , 1] + X[ , 2]); lm(Y[ , 2] ~ X[ , 1] + X[ , 2]) – caracal Nov 14 '10 at 12:26
If it's equivalent, what's the purpose? – Joshua Rosenberg Oct 26 '15 at 23:43

@Brett's response is fine.

If you are interested in describing your two-block structure, you could also use PLS regression. Basically, it is a regression framework which relies on the idea of building successive (orthogonal) linear combinations of the variables belonging to each block such that their covariance is maximal. Here we consider that one block $X$ contains explanatory variables, and the other block $Y$ responses variables, as shown below:

We seek "latent variables" who account for a maximum of information (in a linear fashion) included in the $X$ block while allowing to predict the $Y$ block with minimal error. The $u_j$ and $v_j$ are the loadings (i.e., linear combinations) associated to each dimension. The optimization criteria reads

$$\max_{\mid u_h\mid =1,\mid v_h\mid =1}\text{cov}(X_{h-1}u_h,Yv_h)\quad \big(\equiv \max\text{cov}(\xi_h,\omega_h)\big)$$

where $X_{h-1}$ stands for the deflated (i.e., residualized) $X$ block, after the $h^\text{th}$ regression.

The correlation between factorial scores on the first dimension ($\xi_1$ and $\omega_1$) reflects the magnitude of the $X$-$Y$ link.

-

Multivariate regression is done in SPSS using the GLM-multivariate option.

Put all your outcomes (DVs) into the outcomes box, but all your continuous predictors into the covariates box. You don't need anything in the factors box. Look at the multivariate tests. The univariate tests will be the same as separate multiple regressions.

As someone else said, you can also specify this as a structural equation model, but the tests are the same.

(Interestingly, well, I think it's interesting, there's a bit of a UK-US difference on this. In the UK, multiple regression is not usually considered a multivariate technique, hence multivariate regression is only multivariate when you have multiple outcomes / DVs.)

-
an addition at @Jeremy Miles answer: www-01.ibm.com/support/docview.wss?uid=swg21476743 – Epaminondas Sep 26 '13 at 16:39

I would do this by first transforming the regression variables to PCA calculated variables, and then I would to the regression with the PCA calculated variables. Of course I would store the eigenvectors to be able to calculate the corresponding pca values when I have a new instance I wanna classify.

-
This seems conceptually different than the answer above. I'm still not clear as to how transforming my variables to PCA coefficients allows me to regress on 2+ dependent variables? – Jeff Nov 14 '10 at 20:55
@Jeff this answer is actually conceptually similar to multivariate regression. Here, the suggestion is to do two discrete steps in sequence (i.e., find weighted linear composite variables then regress them); multivariate regression performs the two steps simultaneously. Multivariate regression will be more powerful, as the WLCV's are formed so as to maximize the regression. However, the two-step procedure may provide more clarity regarding the process, or be otherwise preferable for the researcher. – gung Jan 4 '12 at 22:17
@gung Hmm this sort of makes sense to me if you do PCA on the dependent variables and then regress on only the first principal component... But then you're still throwing out a lot of variance. – Jeff Jan 24 '15 at 19:11
@Jeff, the PCs are orthogonal. You can run independent regressions on each. However, this is less powerful b/c the decomposition isn't quite the same, & the multivariate regression test isn't really a series of regs on the PCs. – gung Jan 24 '15 at 20:54
@gung yes, you can run N independent regressions, but then you end up with N sets of beta coefficients. i don't understand how that solves the problem? – Jeff Jan 24 '15 at 21:25

Did you already come across the term "canonical correlation"? There you have sets of variables on the independent as well as on the dependent side. But maybe there are more modern concepts available, the descriptions I have are all of the eighties/nineties...

-
Canonical correlation is the correlation between factor scores computed from two-block structures, as with CCA or PLS. This is exactly what I described in my response (PLS regression), although PLS is more appropriate than CCA when the variables play an asymmetrical role, which is likely to be the case here. This is because there's an asymmetric deflation process and we work with the covariance instead (with CCA, we deflate both blocks at the same time, and we seek to maximize the correlation, instead of the covariance). – chl Nov 16 '10 at 7:50
@chl: upps- today(end of january) I came back to this question/conversation of mid-november.... Sorry I didn't check earlier - there was something with my courses and then I forgot the stat.exchange... If I've something worth I'll come back next days. – Gottfried Helms Jan 28 '11 at 23:23

As mentionned by caracal, you can use mvtnorm package in R. Assuming you made a lm model (named "model") of one of the response in your model, and called it "model", here is how to obtain the multivariate predictive distribution of several response "resp1","resp2","resp3" stored in a matrix form Y:

library(mvtnorm)
model = lm(resp1~1+x+x1+x2,datas) #this is only a fake model to get
#the X matrix out of it
Y = as.matrix(datas[,c("resp1","resp2","resp3")])
X =  model.matrix(delete.response(terms(model)),
data, model\$contrasts)
XprimeX  = t(X) %*% X
XprimeXinv = solve(xprimex)
hatB =  xprimexinv %*% t(X) %*% Y
A = t(Y - X%*%hatB)%*% (Y-X%*%hatB)
F = ncol(X)
M = ncol(Y)
N = nrow(Y)
nu= N-(M+F)+1 #nu must be positive
C_1 =  c(1  + x0 %*% xprimexinv %*% t(x0)) #for a prediction of the factor setting x0 (a vector of size F=ncol(X))
varY = A/(nu)
postmean = x0 %*% hatB
nsim = 2000
ysim = rmvt(n=nsim,delta=postmux0,C_1*varY,df=nu)


Now, quantiles of ysim are beta-expectation tolerance intervals from the predictive distribution, you can of course directly use the sampled distribution to do whatever you want.

To answer Andrew F., degrees of freedom are hence nu=N-(M+F)+1... N being the # of observations, M the # of responses and F the # of parameters per equation model. nu must be positive.

(You may want to read my work on in this document :-) )

-

It's called structural equation model or simultaneous equation model.

-
I could be wrong, but I don't think this is the same thing. From SEM graphs that I've seen, it looks like SEM uses multiple regression equations to determine the values of latent factors, and then another regression is run on the value of those latent factors to determine a higher-order factor. Maybe this is wrong, but I've never seen an SEM graph that links several IVs to multiple DVs-- everything is hierarchical. – Jeff Nov 14 '10 at 20:59
Figure 8 in this paper: biomedcentral.com/1471-2288/3/27 You can do it, but there's little point. It's the same as MANOVA. – Jeremy Miles Mar 25 '13 at 16:15

## protected by Glen_b♦Oct 8 '15 at 22:03

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).