# A multiple regression vs combination of simple regressions using one variable at a time

I am trying to understand whether there is a statistical foundation for a formulation of a multiple regression problem as a combination of multiple simple regressions. Say we have a response variable value $$y_1$$ (a single data point of $$y$$ for the sake of example) and 3 independent variables with observed values $$x_{11}$$, $$x_{12}$$, $$x_{13}$$ for that sample. In a standard multiple regression, we aim to find $$w_1$$, $$w_2$$, $$w_3$$ triple that minimizes below error:

$$(y_1-x_{11}w_1-x_{12}w_2-x_{13}w_3)^2$$

And here is a formulation of the same problem a little differently, such that it minimizes below error instead:

$$(y_1-x_{11}w_1)^2 + (y_1-x_{12}w_2)^2+(y_1-x_{13}w_3)^2$$

The latter formulation is essentially the sum of the errors from 3 one-variable regression problems considering one variable at a time in a simple linear regression.

I am wondering whether there is a method which formulates a multi-variable regression problem using the latter error formula. If not, why is that, i.e., what is wrong with that latter formulation?

• I think your formulation with two samples is confusing. Can you not ask the question with just one sample? If it doesn't work with one sample (and it certainly doesn't), then there's little reason it should work with two samples. There's a related theorem which may be of interest to you though. en.wikipedia.org/wiki/… Apr 17, 2020 at 3:59
• Simplified now. Apr 17, 2020 at 4:15
• The second method would compute 3 different values, if you wanted to combine them in a smart way(using weights) that is no different from least squares. If you average them out, you fail to capture information that exists in the model using multiple variables. Apr 17, 2020 at 5:39

You'll notice the second term of the first equation is missing from the second equation. This is exactly why we prefer equation 1 over 2. By creating 2 (or 3) separate simple models instead of the full model, we are losing variance that might be explained by the interactions between $$x_1$$ and $$x_2$$ and their weights. (This interaction is not to be confused with adding the term $$x_1*x_2$$ to the original model). There is also a $$y^2$$ missing from equation 1. I imagine it can be shown that $$y^2\ge 2*w_1*w_2*x_1*x_2$$, and we are never decreasing (and almost always increasing) our error when we do not use the full model (Someone should prove this, though, I can't see it yet. It might also hold with a strict inequality.). Either way, exactly how damaging this is will depend on the data and is unknown to us, at least in a least-squares sense.