9
$\begingroup$

In the univariate regression settings, we try to model

$$y = X\beta +noise$$

where $y \in \mathbb{R}^n$ a vector of $n$ observations and $X \in \mathbb{R}^{n \times m}$ the design matrix with $m$ predictors. The solution is $\beta_0 = (X^TX)^{-1}Xy$.

In the multivariate regression settings, we try to model

$$Y = X\beta +noise$$

where $y \in \mathbb{R}^{n \times p}$ is a matrix of $n$ observations and $p$ different latent variables. The solution is $\beta_0 = (X^TX)^{-1}XY$.

My question is how is that different than performing $p$ different univariate linear regression? I read here that in the latter case we take into consideration correlation between the dependent variables, but I don't see it from the math.

$\endgroup$
  • 1
    $\begingroup$ See the Frisch-Waugh-Lovell theorem. $\endgroup$ – amorfati Dec 14 '17 at 10:53
  • 1
    $\begingroup$ @amorfati: So if I understand correctly, they are the same. Why do people treat them differently? $\endgroup$ – Roy Dec 14 '17 at 11:13
6
$\begingroup$

In the setting of classical multivariate linear regression, we have the model:

$$Y = X \beta + \epsilon$$

where $X$ represents the independent variables, $Y$ represents multiple response variables, and $\epsilon$ is an i.i.d. Gaussian noise term. Noise has zero mean, and can be correlated across response variables. The maximum likelihood solution for the weights is equivalent to the least squares solution (regardless of noise correlations) [1][2]:

$$\hat{\beta} = (X^T X)^{-1} X^T Y$$

This is equivalent to independently solving a separate regression problem for each response variable. This can be seen from the fact that the $i$th column of $\hat{\beta}$ (containing weights for the $i$th output variable) can be obtained by multiplying $(X^T X)^{-1} X^T$ by the $i$th column of $Y$ (containing values of the $i$th response variable).

However, multivariate linear regression differs from separately solving individual regression problems because statistical inference procedures account for correlations between the multiple response variables (e.g. see [2],[3],[4]). For example, the noise covariance matrix shows up in sampling distributions, test statistics, and interval estimates.

Another difference emerges if we allow each response variable to have its own set of covariates:

$$Y_i = X_i \beta_i + \epsilon_i$$

where $Y_i$ represents the $i$th response variable, and $X_i$ and $\epsilon_i$ represents its corresponding set of covariates and noise term. As above, the noise terms can be correlated across response variables. In this setting, there exist estimators that are more efficient than least squares, and cannot be reduced to solving separate regression problems for each response variable. For example, see [1].

References

  1. Zellner (1962). An efficient method of estimating seemingly unrelated regressions and tests for aggregation bias.
  2. Helwig (2017). Multivariate linear regression [Slides]
  3. Fox and Weisberg (2011). Multivariate linear models in R. [Appendix to: An R Companion to Applied Regression]
  4. Maitra (2013). Multivariate Linear Regression Models. [Slides]
$\endgroup$
  • 1
    $\begingroup$ Thanks, it is clearer now. Do you have a reference for this formulation? I've only encountered the least square form. Also, do you know a Python package the implements that? $\endgroup$ – Roy Dec 14 '17 at 13:13
  • 1
    $\begingroup$ Second the reference request. Does one take the correlation to be just the covariance of the outcomes, or does one learn some sort if conditional covariance? $\endgroup$ – generic_user Dec 14 '17 at 13:39
  • $\begingroup$ I'm not 100% sure that @user20160 was referring to these but I think what they had in mind was estimating equations/generalized estimating equations. EE/GEE are consistent when the covariance structure is misspecified and you can also set the expected covariance structure. However, these models are iteratively estimated as opposed to OLS with a closed form. You should be able to estimate GEE/EE in Python but I do not know the packages. $\endgroup$ – iacobus Dec 14 '17 at 16:13
  • 1
    $\begingroup$ @Roy I rewrote the answer and added references. My original post was assuming the case that is now the last paragraph of the revised post. I'll try to add more detail later. $\endgroup$ – user20160 Dec 15 '17 at 11:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.