Why do we need multivariate regression (as opposed to a bunch of univariate regressions)?

Question

I just browsed through this wonderful book: Applied multivariate statistical analysis by Johnson and Wichern. The irony is, I am still not able to understand the motivation for using multivariate (regression) models instead of separate univariate (regression) models. I went through stats.statexchange posts 1 and 2 that explain (a) difference between multiple and multivariate regression and (b) interpretation of multivariate regression results, but I am not able to tweak out the use of multivariate statistical models from all the information I get online about them.

My questions are:

1. Why do we need multivariate regression? What is the advantage of considering outcomes simultaneously rather than individually, in order to draw inferences.
2. When to use multivariate models and when to use multiple univariate models (for multiple outcomes).
3. Take an example given in the UCLA site with three outcomes: locus of control, self-concept, and motivation. With respect to 1. and 2., can we compare the analysis when we do three univariate multiple regression versus one multivariate multiple regression? How to justify one over another?
4. I haven't come across many scholarly papers that utilize multivariate statistical models. Is this because of the multivariate normality assumption, the complexity of model fitting/interpretation or any other specific reason?

Answer 1

Did you read the full example on the UCLA site that you linked?

Regarding 1:
Using a multivariate model helps you (formally, inferentially) compare coefficients across outcomes.
In that linked example, they use the multivariate model to test whether the write coefficient is significantly different for the locus_of_control outcome vs for the self_concept outcome. I'm no psychologist, but presumably it's interesting to ask whether your writing ability affects/predicts two different psych variables in the same way. (Or, if we don't believe the null, it's still interesting to ask whether you have collected enough data to demonstrate convincingly that the effects really do differ.)
If you ran separate univariate analyses, it would be harder to compare the write coefficient across the two models. Both estimates would come from the same dataset, so they would be correlated. The multivariate model accounts for this correlation.

Also, regarding 4:
There are some very commonly-used multivariate models, such as Repeated Measures ANOVA . With an appropriate study design, imagine that you give each of several drugs to every patient, and measure each patient's health after every drug. Or imagine you measure the same outcome over time, as with longitudinal data, say children's heights over time. Then you have multiple outcomes for each unit (even when they're just repeats of "the same" type of measurement). You'll probably want to do at least some simple contrasts: comparing the effects of drug A vs drug B, or the average effects of drugs A and B vs placebo. For this, Repeated Measures ANOVA is an appropriate multivariate statistical model/analysis.

Answer 2

Think about all the false and sometimes dangerous conclusions that come from simply multiplying probabilities, thinking events are independent. Because of all the built in redundant safeguards, we put into our nuclear power plants experts using the independence assumption told us that the chance of a major nuclear accident was infinitesimal. But as we saw at Three Mile Island, humans make correlated errors especially when they are in a panic because of one initial error which quickly can compound itself. It might be difficult to construct a realistic multivariate model that characterizes human behavior but realizing the effect of a horrible model (independent errors) is clear.

There are many other examples possible. I will take the Challenger Shuttle disaster as another possible example. The question was whether or not to launch under low temperature conditions. There was some data to suggest that the o-rings could fail at low temperatures. But there was not a lot of data from passed missions to make it clear how high the risk was. NASA has always been concerned with the safety of the astronauts and many redundancies were engineered into the space craft and launch vehicles to make the missions safe.

Yet prior to 1986 there were some system failures and near failures probably due to not identifying all possible failure modes (a difficult task). Reliability modeling is a difficult business. But that is another story. In the case of the the shuttle the manufacturer of the o-rings (Morton Thiokol) had done some testing of the o-rings that indicated the possibility of failure at low temperature.

But the data on a limited number of missions did show some relationship between temperature and failure but because redundancy led some administrators to think multiple o-ring failures would not happen, they put pressure on NASA to launch.

Of course there were many other factors that led to the decision. Remember how President Reagan was so anxious to put a teacher in space so as to demonstrate that it was now safe enough that ordinary people who were not astronauts could safely travel on the shuttle. So political pressure was another big factor affecting the decision. In this case with enough data and a multivariate model the risk could have been better demonstrated. NASA use to try to err on the side of caution. In this case case putting off the launch for a few days until the weather warmed up in Florida would have been prudent.

Post-disaster commissions, engineers, scientist and statisticians did a great deal of analysis and papers were published. Their views may differ from mine. Edward Tufte showed in one of his series of books on graphics that good graphics might have been more convincing. But in the end although these analyses all have merit I think the politics would have still won out.

The moral of these stories is not that these disasters motivated the use of multivariate methods but rather that poor analyses that ignored dependence sometimes lead to gross underestimates of risk. This can lead to overconfidence that can be dangerous. As jwimberley pointed out in the first comment to this thread "Separate univariate models ignore correlations."

Answer 3

Consider this quote from p. 36 of Darcy Olsen's book The Right to Try [1]:

But about sixteen weeks after the [eteplirsen] infusions began, Jenn started noticing changes in [her son] Max. "The kid stopped wanting to use his wheelchair," she says. A few weeks later, he was asking to play outside — something he had not done in years. Then Max started regaining his fine motor skills. He was able to open containers again — a skill he had lost as his [Duchenne muscular dystrophy] had progressed.

Max's mother Jenn is building a coherent picture of his improvement, by pulling together evidence from multiple outcomes that individually might be dismissed as 'noise', but that together are quite compelling. (This evidence synthesis principle is part of the reason pediatricians as a rule never dismiss a parent's instinctive inferences that "something is wrong with my child". Parents have access to a 'multivariate longitudinal analysis' of their kids far richer than the 'oligovariate' cross-sectional analysis accessible to a clinician during a single, brief clinical encounter.)

Abstracting away from the particular case of eteplirsen, consider a hypothetical situation where only a small fraction of study subjects were benefitting from an experimental therapy, let's say because of some shared genetic factor not yet known to science. It's quite possible that for those few subjects, a statistical argument corresponding to Jenn's multivariate story could clearly identify them as 'responders', whereas multiple separate analyses of the faint signals contained in individual outcomes would each yield $p>0.05$, driving a 'null' summative conclusion.

Achieving such evidence synthesis is the core rationale for multivariate outcomes analysis in clinical trials. Statistical Methods in Medical Research had a special issue a few years back [2] devoted to 'Joint Modeling' of multivariate outcomes.

1. Olsen, Darcy. The Right to Try: How the Federal Government Prevents Americans from Getting the Life-Saving Treatments They Need. First edition. New York, NY: Harper, an imprint of HarperCollins Publishers, 2015.
2. Rizopoulos, Dimitris, and Emmanuel Lesaffre. “Introduction to the Special Issue on Joint Modelling Techniques.” Statistical Methods in Medical Research 23, no. 1 (February 1, 2014): 3–10. doi:10.1177/0962280212445800.

Answer 4

Let's make a simple analogy, since that's all I can really try to contribute. Instead of univariate versus multivariate regression, let's consider univariate (marginal) versus multivariate (joint) distributions. Say I have the following data and I want to find "outliers". As a first approach, I might use the two marginal ("univariate") distributions and draw lines at the lower 2.5% and upper 2.5% of each independently. Points falling outside of the resulting lines are considered to be outliers.

But two things: 1) what do we think of points that are outside of the lines for one axis but inside of the lines for the other axis? Are they "partial outliers" or something? And 2) the resulting box doesn't look like it's really doing what we want. The reason is, of course, the the two variables are correlated, and what we intuitively want is to find outliers that are unusual considering the variables in combination.

In this case, we look at the joint distribution, and I've color-coded the points by whether their Mahalanobis distance from the center is within the upper 5% or not. The black points look much more like outliers, even though some outliers lie within both sets of green lines and some non-outliers (red) lie outside of both sets of green lines.

In both cases, we're delimiting the 95% versus the 5%, but the second technique accounts for the joint distribution. I believe multivariate regression is like this, where you substitute "regression" for "distribution". I don't totally get it, and have had no need (that I understand) to do multivariate regression myself, but this is the way I think about it.

[The analogy has issues: the Mahalanobis distance reduces two variables to a single number -- something like the way a univariate regregression takes a set of independent variables and can, with the right techniques, take into account covariances among the independent variables, and results in a single dependent variable -- while a multivariate regression results in multiple dependent variables. So it's sort-of backwards, but hopefully forwards-enough to give some intuition.]

Answer 5

1) Nature isn't always simple. In fact, most phenomena (outcome) we study depend on multiple variables, and in a complex manner. An inferential model based on one variable at a time will most likely have a high bias.

2) Univariate models are the simplest model you can build, by definition. It's fine if you are investigating a problem for the first time, and you want to grasp its single, most essential feature. But if you want a deeper understanding of it, an understanding you can actually leverage because you trust what you are doing, you would use multivariate analyses. And among the multivariate ones, you should prefer the ones that do understand correlation patterns, if you care about model accuracy.

3) Sorry no time to read this one.

4) Papers using multivariate techniques are very common these days - even extremely common in some fields. At the CERN experiments using the Large Hadron Collider data (to take an example from particle physics) more than half the hundreds of papers published each year use multivariate techniques in one way or another

https://inspirehep.net/search?ln=en&ln=en&p=find+cn+cms+&of=hb&action_search=Search&sf=earliestdate&so=d&rm=&rg=25&sc=0

Answer 6

My answer depends on what you want to do with the regression. If you are trying to compare the effect of different coefficients, then regression may not be the right tool for you. If you are trying to make predictions using different coefficients that you have proven are independent, then maybe multiple regression is what you should use.

Are the factors correlated? If so, a multivariate regression can give you a bad model and you should use a method like VIFs or ridge regression to trim cross-correlations. You should not compare coefficients until the cross-correlated factors are eliminated. Doing so will lead to disaster. If they are not cross-correlated, then multivariate coefficients should be as comparable as univariate coefficients, and this should not be surprising.

The outcome might also depend on the software package you are using. I am not joking. Different software packages have different methods for calculating multivariate regression. (Don't believe me? Check out how the standard R regression package calculates R² with and without forcing the origin as the intercept. Your jaw should hit the floor.) You need to understand how the software package is performing the regression. How is it compensating for cross-correlations? Is it performing a sequential or matrix solution? I've had frustrations with this in the past. I suggest performing your multiple regression on different software packages and see what you get.

Another good example here:

Note that in this equation, the regression coefficients (or B coefficients) represent the independent contributions of each independent variable to the prediction of the dependent variable. Another way to express this fact is to say that, for example, variable X1 is correlated with the Y variable, after controlling for all other independent variables. This type of correlation is also referred to as a partial correlation (this term was first used by Yule, 1907). Perhaps the following example will clarify this issue. You would probably find a significant negative correlation between hair length and height in the population (i.e., short people have longer hair). At first this may seem odd; however, if we were to add the variable Gender into the multiple regression equation, this correlation would probably disappear. This is because women, on the average, have longer hair than men; they also are shorter on the average than men. Thus, after we remove this gender difference by entering Gender into the equation, the relationship between hair length and height disappears because hair length does not make any unique contribution to the prediction of height, above and beyond what it shares in the prediction with variable Gender. Put another way, after controlling for the variable Gender, the partial correlation between hair length and height is zero. http://www.statsoft.com/Textbook/Multiple-Regression

There are so many pitfalls using multiple regression that I try to avoid using it. If you were to use it, be very careful with the outcomes and double check them. You should always plot the data visually to verify the correlation. (Just because your software program said there was no correlation, doesn't mean there isn't one. Interesting Correlations) Always check your results against common sense. If one factor shows a strong correlation in a univariate regression, but none in multivariate, you need to understand why before sharing the results(the gender factor above is a good example).