What are the major differences between the parameter estimation of a simple linear regression vs multiple?

Question

This is the standard technique that I have used to calculate the parameter of a single linear regression problem with one independent variable. How does the parameter estimation problem differ in multiple linear regression with 'k' independent variables. I assume I'd treat each independent variable in a multiple linear regression problem separately, which would then give me 'k' different sub simple linear regression problems, and then I'd add them parameters to give me my final model. I am requesting to be corrected or improved upon.

Answer 1

Suppose the following multiple linear regression model:

$\mathbf{y} = \mathbf{X\beta}+\mathbf{\epsilon}$, where I use bold lower case letters to represent vectors ($\mathbf{y},\mathbf{\beta},\mathbf{\epsilon}$) and capitals with bold to represent matrices ($\mathbf{X}$).

Ordinary Least Squares seeks to minimize the residuals of the model (the $\mathbf{\epsilon}$ values). The estimates of $\mathbf{\beta}$ can be obtained using the closed form solution (Add a column of 1's to $\mathbf{X}$ to include an intercept):

$\hat{\boldsymbol\beta} = (\mathbf{X}^{\rm T}\mathbf{X})^{-1} \mathbf{X}^{\rm T}\mathbf{y} = \big(\, \tfrac{1}{n}{\textstyle\sum} \mathbf{x}_i \mathbf{x}^{\rm T}_i \,\big)^{-1} \big(\, \tfrac{1}{n}{\textstyle\sum} \mathbf{x}_i y_i \,\big).$

It may still not be obvious that this is different than estimating each parameter in the vector $\mathbf{\beta}$ separately with a simple linear regression and including them in a final model. Maybe this example can provide some intuition: suppose you have $y$ and $\mathbf{x}_1$ and $\mathbf{x}_2$ and also suppose that $\mathbf{x}_1=\mathbf{x}_2$; in other words they are perfectly correlated. If you estimate the multiple linear regression model for $\mathbf{x}_1$ and $\mathbf{x}_2$ together, or the simple linear regression model for each of them separately, you will still have the same sort of objective function - one that seeks to minimize the residuals.

However, you will obtain different results, and the simple linear regression will probably be wrong. Suppose you estimate the model for $\mathbf{x}_1$ and get $\mathbf{\hat{\beta}}_1=5$. This gives use that for one unit change in $\mathbf{x}_1$, we have a five unit change in $\mathbf{y}$. Since $\mathbf{x}_1=\mathbf{x}_2$, a simple linear regression of $\mathbf{y}$ on $\mathbf{x}_2$ would also give $\mathbf{\hat{\beta}}_2=5$.

If you add both of these parameters up, as you say in your question you would get the model

$\mathbf{y} = 5\mathbf{x}_1+5\mathbf{x}_2$, but since $\mathbf{y}$ on $\mathbf{x}_2$ you are doubling their impact (they do not contain different information). This model will not have minimal residuals, so it would not be the optimal multiple linear regression model. Instead, if you run multiple linear regression, the coefficients will be more balanced between $\mathbf{x}_1$ and $\mathbf{x}_2$ to reflect the fact that they share the same information.

A caveat: if the goal is either inference or predictive modeling, the performance of OLS estimates can be poor if multicollinearity is present (as it is in my example), unless the sample size is large.

So multiple regression can still work in the example I gave, especially if the sample size is large, but you should be careful of multicollinearity nonetheless. That said, the results from estimating separate simple linear regressions and adding the parameters to get a final model are wrong. The purpose of the example is to provide some intuition into that rather than to be completely realistic.

Answer 2

One algorithm that operates similarly to a series of "simple" linear regressions is by using the QR decomposition for X, in particular the gram-schmidt algorithm. Now it is well known that the OLS estimate are given as $\hat{\beta}=(X^TX)^{-1} X^Ty $ where $X $ has $n$ rows and $ k+1 $ columns, the ith row being $ x_i=(1, x_{1i},\dots, x_{ki}) $. Using the QR method means that $ X=QR $ and $hat {\beta}=(R^TR)^{-1} R^TQ^Ty =R^{-1} Q^Ty$. Wikipedia has some info on how the algorithm works. Basically you "add a variable" to your regression one at a time, but you work with "current residuals" instead of your response, and covariates also get adjusted for variables already in the model.

You can show this will simple OLS with $ k=1 $. Using the wikipedia article, you have $ u_1=1_n, e_1=1_n\frac {1}{\sqrt{n} }$ giving $ R_{11}=\sqrt {n} $. This is the "intercept only regression, noting that the parameter is $ b_0=R_{11}^{-1} Q_{1}^Ty=(\sqrt {n})^{-1}\frac {1}{\sqrt {n}}1_n^Ty=\overline {y} $ giving the standard estimate when no regression variables are used. Now proceed to step 2 of the algorithm, and $ u_2 =x-e_1^Txe_1 =x-1_n^Tx\frac {1}{n}=x-\overline {x} 1_n $ and $ e_2 =(x-\overline {x} 1_n)S_{xx}^{-1} $. Note that $ u_2 $ is the residuals from regressing $1_n $ on $ x $. We have $ R_{12}=e_1^Tx=\sqrt {n}\overline {x} $ and $ R_{22}=1 $. Now the inverse of $ R $ is given as $ R^{-1}_{11}=\frac {1}{\sqrt {n}} $ as before, and $ R^{-1}_{12}=-\overline {x} $, and finally $ R^{-1}_{22}=1 $. We also get a second term to $ Q^Ty $ and this is equal to the ols slope. That is $ Q^Ty=(\overline {y}, b_1)^T $. Multiplying this by $ R^{-1} $ gives you the adjustment for the intercept. To add a third variable, say $ z $ you create $ u_3=z-e_2^Tze_2-e_1^Tze_1 $. Now these are the residuals from and ols fit to z using x and an intercept. Hopefully you can see the pattern from here...