96
$\begingroup$

Are multiple and multivariate regression really different? What is a variate anyways?

$\endgroup$
0

7 Answers 7

70
$\begingroup$

Very quickly, I would say: 'multiple' applies to the number of predictors that enter the model (or equivalently the design matrix) with a single outcome (Y response), while 'multivariate' refers to a matrix of response vectors. Cannot remember the author who starts its introductory section on multivariate modeling with that consideration, but I think it is Brian Everitt in his textbook An R and S-Plus Companion to Multivariate Analysis. For a thorough discussion about this, I would suggest to look at his latest book, Multivariable Modeling and Multivariate Analysis for the Behavioral Sciences.

For 'variate', I would say this is a common way to refer to any random variable that follows a known or hypothesized distribution, e.g. we speak of gaussian variates $X_i$ as a series of observations drawn from a normal distribution (with parameters $\mu$ and $\sigma^2$). In probabilistic terms, we said that these are some random realizations of X, with mathematical expectation $\mu$, and about 95% of them are expected to lie on the range $[\mu-2\sigma;\mu+2\sigma]$ .

$\endgroup$
2
  • 3
    $\begingroup$ Even coursera.org/learn/machine-learning/home/week/2 uses the term multivariate regression instead of multiple regression… $\endgroup$ Commented Oct 28, 2015 at 2:32
  • 2
    $\begingroup$ I think the same confusion arises with people using the term GLM for General Linear Model (e.g., in neuroimaging studies) vs. Generalised Linear Model. I have seen many instances of "multivariate logistic regression" where there's only one outcome, and I don't think this matters so much as long as the term is clearly defined by the author. $\endgroup$
    – chl
    Commented Oct 28, 2015 at 10:38
75
$\begingroup$

Here are two closely related examples which illustrate the ideas. The examples are somewhat US centric but the ideas can be extrapolated to other countries.

Example 1

Suppose that a university wishes to refine its admission criteria so that they admit 'better' students. Also, suppose that a student's grade Point Average (GPA) is what the university wishes to use as a performance metric for students. They have several criteria in mind such as high school GPA (HSGPA), SAT scores (SAT), Gender etc and would like to know which one of these criteria matter as far as GPA is concerned.

Solution: Multiple Regression

In the above context, there is one dependent variable (GPA) and you have multiple independent variables (HSGPA, SAT, Gender etc). You want to find out which one of the independent variables are good predictors for your dependent variable. You would use multiple regression to make this assessment.

Example 2

Instead of the above situation, suppose the admissions office wants to track student performance across time and wishes to determine which one of their criteria drives student performance across time. In other words, they have GPA scores for the four years that a student stays in school (say, GPA1, GPA2, GPA3, GPA4) and they want to know which one of the independent variables predict GPA scores better on a year-by-year basis. The admissions office hopes to find that the same independent variables predict performance across all four years so that their choice of admissions criteria ensures that student performance is consistently high across all four years.

Solution: Multivariate Regression

In example 2, we have multiple dependent variables (i.e., GPA1, GPA2, GPA3, GPA4) and multiple independent variables. In such a situation, you would use multivariate regression.

$\endgroup$
2
  • 7
    $\begingroup$ There's always one that properly answers the question with examples :) $\endgroup$ Commented Aug 30, 2017 at 2:44
  • 3
    $\begingroup$ 100% the best answer that you can actually understand $\endgroup$
    – Alvis
    Commented Apr 16, 2018 at 14:08
47
$\begingroup$

Simple regression pertains to one dependent variable ($y$) and one independent variable ($x$): $y = f(x)$

Multiple regression (aka multivariable regression) pertains to one dependent variable and multiple independent variables: $y = f(x_1, x_2, ..., x_n)$

Multivariate regression pertains to multiple dependent variables and multiple independent variables: $y_1, y_2, ..., y_m = f(x_1, x_2, ..., x_n)$. You may encounter problems where both the dependent and independent variables are arranged as matrices of variables (e.g. $y_{11}, y_{12}, ...$ and $x_{11}, x_{12}, ...$), so the expression may be written as $Y = f(X)$, where capital letters indicate matrices.

Further reading:

$\endgroup$
5
  • 1
    $\begingroup$ I understand the definition. But what is the effect of treating a multi-variate regression as a system of uni-variate regressions? $\endgroup$
    – LKS
    Commented Aug 3, 2016 at 19:30
  • $\begingroup$ @LKS: You may want to ask that in a completely separate question. $\endgroup$ Commented Oct 5, 2016 at 21:10
  • 2
    $\begingroup$ stats.stackexchange.com/questions/254254/… $\endgroup$
    – KarthikS
    Commented Jan 26, 2018 at 23:43
  • $\begingroup$ Did the answer in the Quora referring to this page? :P $\endgroup$
    – user167493
    Commented Nov 6, 2018 at 6:31
  • $\begingroup$ This is a very clear explanation, but is it possible that some people use the terms differently? For example, that "multivariate regression" pertains to multiple dependent variables and a single dependent variable? I'm trying to make sense of the term "multivariate multiple regression" and how it differs from multivariate regression as defined here. $\endgroup$
    – Liam
    Commented Feb 19, 2023 at 15:21
11
$\begingroup$

I think the key insight (and differentiator) here aside from the number of variables on either side of the equation is that for the case of multivariate regression, the goal is to utilize the fact that there is (generally) correlation between response variables (or outcomes). For example, in a medical trial, predictors might be weight, age, and race, and outcome variables are blood pressure and cholesterol. We could, in theory, create two "multiple regression" models, one regressing blood pressure on weight, age, and race, and a second model regressing cholesterol on those same factors. However, alternatively, we could create a single multivariate regression model that predicts both blood pressure and cholesterol simultaneously based on the three predictor variables. The idea being that the multivariate regression model may be better (more predictive) to the extent that it can learn more from the correlation between blood pressure and cholesterol in patients.

$\endgroup$
1
  • 1
    $\begingroup$ Great point. I was wondering if multivariate regression can be done with R. Using Manova, I am able to do multivariate ANOVA, but not able to get coefficients like univariate regression. $\endgroup$
    – KarthikS
    Commented Jan 26, 2018 at 23:45
4
$\begingroup$

In multivariate regression there are more than one dependent variable with different variances (or distributions). The predictor variables may be more than one or multiple. So it is may be a multiple regression with a matrix of dependent variables, i. e. multiple variances. But when we say multiple regression, we mean only one dependent variable with a single distribution or variance. The predictor variables are more than one. To summarise multiple refers to more than one predictor variables but multivariate refers to more than one dependent variables.

$\endgroup$
1
$\begingroup$

There is no difference. This is because the maximum likelihood solution of the parameters of the joint problem $Y = W^T φ(x)$ with K target variables decouples to K independent regression problems, assuming a conditional distribution of the target vector to be an isotropic Gaussian of the form $p(t|φ(x),W, β) = N (t|W^T φ(x), β^{-1} I)$. Refer to section '3.1.5 Multiple outputs' from the book 'Pattern Recognition and Machine Learning', Bishop for details.

$\endgroup$
2
  • $\begingroup$ While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. - From Review $\endgroup$ Commented Apr 17, 2023 at 9:26
  • $\begingroup$ Thank you for the suggestion, hope it is now better. $\endgroup$
    – ahmed musa
    Commented Apr 17, 2023 at 13:43
0
$\begingroup$

There ain’t no difference between multiple regression and multivariate regression in that, they both constitute a system with 2 or more independent variables and 1 or more dependent variables. As long as the outcome doesn’t depend on lag obs or a single predictor, it’s called multiple or multivariate regression otherwise it is termed univariate regression.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.