# Stepwise regression seems better than LASSO, why?

I am trying to figure out why I keep finding better results using stepwise regression than with LASSO while there are hundreds of posts and papers stating the opposite.

To explain a bit better : I got a pool of 20 variables I want to select from and about 150 other variables I am enforcing in the model. (It is in an association study context, the 20 selectable variables being genetic markers and the rest being PCA components allowing control of the kinship between individuals.)

The 20 variables are quite correlated as shown below :

I am trying to get a subset of the markers that still explains the response variable in a 'good enough' manner. For that I used two methods a forward/backward regression and a LASSO regression.

I am a bit puzzled by the results :

$\begin{array}{r|c|c} & Stepwise & LASSO \\ \hline Number\ of\ variables\ selected & 10 & 15 \\ Correlation\ fitted\ vs.\ observed\ values & 0.849 & 0.846 \\ MSE & 323 & 330 \end{array}$

I was not expecting LASSO to absolutely select less variables than the stepwise algorithm but I did expect that the end results would be better. How can I explain those ? Are the criterion I am using not appropriate or is it because I am using fitted values and not predicted values of additional data, or is it a totally different matter ?

Note that I used the shrinked model to get the fitted values so it is not the same problem as here

• Your table represents how well you fit your present data set. Have you run any tests related to the generalizability of your model to other samples, like repeating your procedures on multiple bootstrap resamples? That's where the superiority of LASSO is more likely to be seen. Please try that and add those results to your question. – EdM Dec 7 '16 at 10:57
• The MSE and correlation were estimated via a 5-folds CV, sorry if I was not clear. – Riff Dec 7 '16 at 11:56
• Agree with EdM, to get a fair performance evaluation you need to test your model on new data that was not used for selecting and/or fitting the model. – Richard Hardy Dec 7 '16 at 12:08
• A single 5-fold CV using your single set of model parameters is not enough. You need to do many CVs or, perhaps better, multiple bootstrap resamples of your data. Then repeat the entire model building process, including the variable selection steps, on each resample and see how well the model built on the resample fit your original data set. Do this hundreds of times, and combine the results. – EdM Dec 7 '16 at 13:17
• I don't understand how multiple CV would lead to different results. The dataset is very small (~250 individuals) and I can't totally randomize the folds as there are relationship between individuals that needs to be preserved in each fold so that means very few different possible fold. As for bootstrap, I will try but it may prove difficult because of the relatedness problem. – Riff Dec 7 '16 at 13:22

• Thanks for the answer. Since you seem to be familiar with biologic data, I have a follow-up : would you then recommend using less PCA components of the kinship matrix or is essential to get as much of the initial inertia as possible ? (my problem is that I usually model relatedness modelling it as a random effect in a mixed model as $\mathcal{N}(0, \sigma^2K$) so I am not familiar with using principal components as fixed effects) – Riff Dec 7 '16 at 16:12