For predictors $X_1$ and $X_2$, and a response $Y$, two models are fit:
Model 1: Linear regression with $Y$ as response and $X_1$ and $X_2$ as predictors (only the linear terms).
Model 2: A linear regression model is fit with $Y$ as response and $X_1$ as a predictor. Using the residuals $R$ for this model, another model is fit for $R$ against $X_2$.

Apparently, Model 1 and Model 2 are equivalent (fitted values are the same) only when linear terms of $X_1$ and $X_2$ are involved, and $X_1$ is not correlated with $X_2$. How can this be mathematically proved?

Additionally, how does this concept relate to the boosting of regression trees, wherein the updated residuals are used as a response in successive regression trees?

  • $\begingroup$ Can you write it in mathematical notation and provide a source that states equivalence of fitted values? As far as i understood it I don't think it is true $\endgroup$ – Łukasz Grad Mar 4 '17 at 12:50
  • $\begingroup$ @ŁukaszGrad It was mentioned in an explanation for the quiz 7 problem from the Statistical Learning MOOC by Hastie and Tibshirani. Please let me know if that works! $\endgroup$ – Pradnyesh Joshi Mar 4 '17 at 18:15
  • $\begingroup$ This question needs a self study tag. $\endgroup$ – Michael Chernick Mar 4 '17 at 19:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.