3
$\begingroup$

This post is to understand differences between multivariate linear regression models (i.e multiple independent variables predicting multiple dependent variables) and running multiple linear regression models for different dependent variables.

  • When we run multiple linear regression models for different dependent variables, each model will have the weights that corresponding to that specific dependent variable.

  • And when we consider a multivariate model, the coefficients (i.e weights) for each output (dependent variable) in this model are still independently calculated (I believe) for every set of independent variables.

  • This should generate similar values for coefficients for a given dependent variable in a multivariate model and when running an isolated linear regression model.

How does a multivariate model help? Please correct me if I misunderstood anything.

$\endgroup$
2
  • $\begingroup$ Discussion point: Multivariate model will only help in case the independent features are intermediate representations in the bigger model - because in this case the effect of having multiple outputs together has impact on the weights of the intermediate layer as compared to a linear model where the independent features are the first layer. Please let me know if my understanding is correct $\endgroup$ Sep 17, 2023 at 12:08
  • $\begingroup$ The maximum likelihood estimates are identical whether you consider the regressions separately or jointly, assuming you have the same X matrix for all Y variables, and assuming an unstructured covariance matrix, so in that case it does not matter whether you do the regressions separately or jointly. However, if you wish to compare the regression functions for different Y variables, you need to use the multivariate framework, because the standard errors of the differences involve the (conditional on X) covariances between the Y variables. $\endgroup$ Sep 17, 2023 at 21:05

1 Answer 1

0
$\begingroup$

It probably depends on the type of multivariate technique you use, but to answer some of your questions:

When we run multiple linear regression models for different dependent variables, each model will have the weights that corresponding to that specific dependent variable.

Correct. Though remember for each main effect the coefficient represents a partial effect (it is the influence on the dependent variable after controlling for other variables in the regression) and interactions are simply the product of main effects.

And when we consider a multivariate model, the coefficients (i.e weights) for each output (dependent variable) in this model are still independently calculated (I believe) for every set of independent variables.

This isn't necessarily true. Probably the most obvious example of this not being the case is a covariance-based structural equation model (SEM). These use a covariance matrix which approximates a match between an implied model and the data itself. SEMs like these are also routinely checked for global fit, something that is fairly rare or nonexistent in the multiple linear regression landscape (at least in my area), so by definition it particularly requires looking at the relationships as a whole (global fit) rather than it's component pieces (local fit), though the latter is certainly looked at in both MLR and SEM contexts depending on who is doing the analysis.

This should generate similar values for coefficients for a given dependent variable in a multivariate model and when running an isolated linear regression model.

Not necessarily. Using SEM again as an example, once you build a latent variable into the model, its pretty much impossible to disentangle that into an equivalent MLR relationship. Path models will sometimes give you the same estimates (you can literally just build an MLR in lavaan and get the same estimates back as using a lm fit), so in that sense they would be functionally similar.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.