One of the things I've always been confused about is the framework around model selection in the cases where $n$ predictor variables $x_i$'s are correlated.

My thoughts on the different approaches:

  1. Simple OLS by adding each predictor variable $x_i$ and looking at some metric of model fit (like adjusted $R^2$) and stopping at some threshold
  2. OLS but with regularization (either lasso which explicitly will pick the explanatory variables or ridge with some threshold for $\beta_i$ of which terms to keep)
  3. Regressing residuals. For example in the 2 variable case ($x_1$ and $x_2$), which approach would be correct if I wanted to assess incremental effect of including $x_2$ to response variable.
    • Approach (a). Take $y_i = \beta_0 + \beta_1 x_{1,i} + \epsilon_i$, then run $\epsilon_i = \beta_3+\beta_2 x_{2,i} + e_i$ to see how well $x_2$ can predict the residuals (ie the error in just $x_1$ explaining $y$)
    • Approach (b). Take $x_{2,i} = \beta_5+\beta_4 x_{1,i} +\delta_i$ and then running $y_i=\beta_6 + \beta_7 x_{1,i} + \beta_{\delta} \delta_i + d_i$. This is basically regressing ($x_1$) and ($x_2$ not explained by $x_1$)
  4. PCA approach and using top factors to regress against $y_i$

My specific questions:

  1. What are the tradeoffs for (1) vs (2) vs (3) vs (4)? It seems like in practice, I hear of (2) and (4) a lot.
  2. More specifically, which approach between (3a) vs (3b) is correct given the goal of assessing incremental effect of including $x_2$ in regression?

I've read Dealing with correlated regressors and How to deal with high correlation among predictors in multiple regression? and What is the effect of having correlated predictors in a multiple regression model?

  • $\begingroup$ It's probably not meaningful to characterize any of these approaches as "correct." But in terms of being likely to give useful results, (3) is tantamount to (1) and neither is any good for model selection. You can see how arbitrary (1) and (3) are just by changing the order in which you introduce variables into the model. More principled versions of (1) and (3) are called stepwise methods. You can search that term here and will discover most readers think little of them, but they have been extensively used. $\endgroup$ – whuber Jan 12 at 20:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.