2
$\begingroup$

I am trying to understand whether there is a statistical foundation for a formulation of a multiple regression problem as a combination of multiple simple regressions. Say we have a response variable value $y_1$ (a single data point of $y$ for the sake of example) and 3 independent variables with observed values $x_{11}$, $x_{12}$, $x_{13}$ for that sample. In a standard multiple regression, we aim to find $w_1$, $w_2$, $w_3$ triple that minimizes below error:

$(y_1-x_{11}w_1-x_{12}w_2-x_{13}w_3)^2$

And here is a formulation of the same problem a little differently, such that it minimizes below error instead:

$(y_1-x_{11}w_1)^2 + (y_1-x_{12}w_2)^2+(y_1-x_{13}w_3)^2$

The latter formulation is essentially the sum of the errors from 3 one-variable regression problems considering one variable at a time in a simple linear regression.

I am wondering whether there is a method which formulates a multi-variable regression problem using the latter error formula. If not, why is that, i.e., what is wrong with that latter formulation?

$\endgroup$
3
  • $\begingroup$ I think your formulation with two samples is confusing. Can you not ask the question with just one sample? If it doesn't work with one sample (and it certainly doesn't), then there's little reason it should work with two samples. There's a related theorem which may be of interest to you though. en.wikipedia.org/wiki/… $\endgroup$
    – Tim Mak
    Apr 17, 2020 at 3:59
  • $\begingroup$ Simplified now. $\endgroup$
    – user5054
    Apr 17, 2020 at 4:15
  • $\begingroup$ The second method would compute 3 different values, if you wanted to combine them in a smart way(using weights) that is no different from least squares. If you average them out, you fail to capture information that exists in the model using multiple variables. $\endgroup$
    – hxlaclhemy
    Apr 17, 2020 at 5:39

1 Answer 1

1
$\begingroup$

I agree equation 1 is solving a simultaneous system of linear equations, whereas equation 2 is solving them independently. As far as a closed-form solution to equation 2, there may or may not be an answer, I'm not sure. Numerically it's pretty straightforward to do. If you compare the slopes for the three first order models vs. the full model, and as long as the data are selected carefully, there will be a difference. This difference can be shown formally:

First, let's make our lives easier by dropping down to 2 dimensions. Next, if you expand the two equations (assuming I used mathHandBook.com correctly) they are not identical, but very similar.

equation1 equation2

You'll notice the second term of the first equation is missing from the second equation. This is exactly why we prefer equation 1 over 2. By creating 2 (or 3) separate simple models instead of the full model, we are losing variance that might be explained by the interactions between $x_1$ and $x_2$ and their weights. (This interaction is not to be confused with adding the term $x_1*x_2$ to the original model). There is also a $y^2$ missing from equation 1. I imagine it can be shown that $y^2\ge 2*w_1*w_2*x_1*x_2$, and we are never decreasing (and almost always increasing) our error when we do not use the full model (Someone should prove this, though, I can't see it yet. It might also hold with a strict inequality.). Either way, exactly how damaging this is will depend on the data and is unknown to us, at least in a least-squares sense.

$\endgroup$
3
  • $\begingroup$ Hey @Rob G thanks for the answer. A followup question: I heard from someone that the second formulation (three simple regressions) with a proper regularization on each term might be less prone to overfitting in the very high-dimensional case where number of variables is too large compared to the number of samples (say 20K variables and 50 samples), although they did not explain why. Any thoughts on that? $\endgroup$
    – user5054
    Apr 18, 2020 at 4:52
  • $\begingroup$ May be obvious, but they are also implying that the regularization is used on the first formulation? Because regularizing vs. not regularizing specifically helps with overfitting. Assuming the obvious, I do not have experience with that many variables, so I'm not sure. By samples, do you mean cases/observations/rows? 50 observations per variable or 50 cases total? If the latter, I'd be highly dubious of using that many variables with so few observations. If the former, it seems reasonable. The difficulty is the subsequent interpretation. If we're only concerned with prediction, it could work. $\endgroup$
    – Rob G
    Apr 19, 2020 at 7:18
  • $\begingroup$ - Yeah sure, they meant when each of the two are regularized properly, not only the latter. - I mean 50 observations per variable. These datasets are pretty common in computational biology. I am not sure what "50 cases in total" would mean, though.. $\endgroup$
    – user5054
    Apr 19, 2020 at 22:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.