A multiple regression vs combination of simple regressions using one variable at a time

Question

I am trying to understand whether there is a statistical foundation for a formulation of a multiple regression problem as a combination of multiple simple regressions. Say we have a response variable value $y_1$ (a single data point of $y$ for the sake of example) and 3 independent variables with observed values $x_{11}$, $x_{12}$, $x_{13}$ for that sample. In a standard multiple regression, we aim to find $w_1$, $w_2$, $w_3$ triple that minimizes below error:

$(y_1-x_{11}w_1-x_{12}w_2-x_{13}w_3)^2$

And here is a formulation of the same problem a little differently, such that it minimizes below error instead:

$(y_1-x_{11}w_1)^2 + (y_1-x_{12}w_2)^2+(y_1-x_{13}w_3)^2$

The latter formulation is essentially the sum of the errors from 3 one-variable regression problems considering one variable at a time in a simple linear regression.

I am wondering whether there is a method which formulates a multi-variable regression problem using the latter error formula. If not, why is that, i.e., what is wrong with that latter formulation?

Answer 1

I agree equation 1 is solving a simultaneous system of linear equations, whereas equation 2 is solving them independently. As far as a closed-form solution to equation 2, there may or may not be an answer, I'm not sure. Numerically it's pretty straightforward to do. If you compare the slopes for the three first order models vs. the full model, and as long as the data are selected carefully, there will be a difference. This difference can be shown formally:

First, let's make our lives easier by dropping down to 2 dimensions. Next, if you expand the two equations (assuming I used mathHandBook.com correctly) they are not identical, but very similar.

You'll notice the second term of the first equation is missing from the second equation. This is exactly why we prefer equation 1 over 2. By creating 2 (or 3) separate simple models instead of the full model, we are losing variance that might be explained by the interactions between $x_1$ and $x_2$ and their weights. (This interaction is not to be confused with adding the term $x_1*x_2$ to the original model). There is also a $y^2$ missing from equation 1. I imagine it can be shown that $y^2\ge 2*w_1*w_2*x_1*x_2$, and we are never decreasing (and almost always increasing) our error when we do not use the full model (Someone should prove this, though, I can't see it yet. It might also hold with a strict inequality.). Either way, exactly how damaging this is will depend on the data and is unknown to us, at least in a least-squares sense.