I know there are several other questions asking for the advantages of multivariate regression over several univariate regressions (e.g. this). I understand that the dependent variables can be correlated and that multivariate regression can account for that.

What I couldn't find, is whether this difference actually plays a role if all I want is to minimize the error (say Euclidean distance) on a particular test set after training. From my understanding, multivariate linear regression can hardly be more general than several univariate linear regressions.

Take a data set of examples where each example is a tuple $<input,~ target > $ consisting of $input \in \mathbb{R}^n $ and $target \in \mathbb{R}^m $. Assume this data set is separated into two disjunct sets: the training set $Train $ and the test set $Test $.

Can you give me an artificial data set, where a multivariate linear (degree 1) regression for $n $ independent variables and $m $ dependent variables performs better than $m $ univariate linear (degree 1) regressions, each of which for $n $ independent variables and only one of the dependent variables? Both procedures are supposed to be generated from a random subset $Train $ consisting of 80% of all examples. Their success is supposed to be determined by the cumulative error over the disjunct subset $Test $ containing the remaining 20% of all examples.

I know that data sets can be constructed to favor any learning procedure over another. Exactly such a data set would help me understand the advantages of multivariate regression from a machine learning perspective.

  • 1
    $\begingroup$ According to this page that was linked in the answer to the quetsion you referenced: stats.idre.ucla.edu/stata/dae/multivariate-regression-analysis "Separate OLS Regressions ...The individual coefficients, as well as their standard errors will be the same as those produced by the multivariate regression." $\endgroup$ – Jonny Lomond Apr 24 at 12:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.