# Fitting models against residuals in the context of linear regression and regression trees

For predictors $X_1$ and $X_2$, and a response $Y$, two models are fit:
Model 1: Linear regression with $Y$ as response and $X_1$ and $X_2$ as predictors (only the linear terms).
Model 2: A linear regression model is fit with $Y$ as response and $X_1$ as a predictor. Using the residuals $R$ for this model, another model is fit for $R$ against $X_2$.

Apparently, Model 1 and Model 2 are equivalent (fitted values are the same) only when linear terms of $X_1$ and $X_2$ are involved, and $X_1$ is not correlated with $X_2$. How can this be mathematically proved?

Additionally, how does this concept relate to the boosting of regression trees, wherein the updated residuals are used as a response in successive regression trees?

• Can you write it in mathematical notation and provide a source that states equivalence of fitted values? As far as i understood it I don't think it is true – Łukasz Grad Mar 4 '17 at 12:50
• @ŁukaszGrad It was mentioned in an explanation for the quiz 7 problem from the Statistical Learning MOOC by Hastie and Tibshirani. Please let me know if that works! – Pradnyesh Joshi Mar 4 '17 at 18:15
• This question needs a self study tag. – Michael Chernick Mar 4 '17 at 19:36