# Nested cross-validation or bootstrapping and non-nested cross validation?

I have 25 correlated independent variables and one dependent variable that is an aggregated score of a Likert scale. I also have 90 samples. I want to do variable selection for linear regression so I am using LASSO. In python I used LASSO CV (coordinate descent). Q1 I can use LASSO AIC/BIC, LASSO CV, LASSO LARS CV is there any reason to pick one over the other? Q2 I tried LASSO CV with nested cross validation as I was told that it will calculate better my hyper parameters since I have only 90 samples but the (3-fold inner/3-fold outer) but the 3 models I get are very different to each other. Do I need to do bootstrapping and do non nested instead?

## 2 Answers

Hi this should probably be a comment and not an answer, but I have written a small R code (see here) that can perform LASSO with nested-CV, in which the inner loop performed LOOCV, which can help make the output models more consistent (so n-fold inner, and k-fold outer). I hope this helps.

• Because I am a newbie could you please explain the difference? Unfortunately I am using Python, can I convert it somehow? – Da Kor Mar 29 '17 at 13:29
• I think I managed to do that in Python. – Da Kor Mar 29 '17 at 19:06
• Great, given the size of your dataset it maybe better to use Leave One Out CV in both inner and outer loops. This should in principle be similar to that of bootstrapping, according to this excellent reference (note that the paper considers logistic regression). – Abbas Mar 29 '17 at 20:20
• That is such a great reference, thank you. It shouldn't matter that the paper considers logistic regression right? I can use it as a reference for linear as well, is that true? Also, I want the predictive value of the model, something like R^2. Can I do that with LOO in outer loop?Can I take the features that appear in every model an the run good old multiple linear regression or this is a no no?It is a bit biased cause I'll retrain to the same data but it makes some sense to me. – Da Kor Mar 30 '17 at 10:44
• I think the same conclusions apply to linear regression. What is the reason for developing the model? Is it for prediction? R^2 is a measure of goodness of fit and not predictive ability. – Abbas Apr 3 '17 at 12:07

Firstly if your models in each CV folds are too different then this is a sign of unstability of model fitting. Either your model is too complex for data or the features are too correlated. You may use a combination of norm 1 and norm 2 regularization to improve the stability of LASSO (known as elastic net regularization https://en.wikipedia.org/wiki/Elastic_net_regularization)

Secondly with your data set you may not really need LASSO for feature selection. LASSO or norm 1 regularization is especially useful when the data set is large. In your case you can easily search for the best feature set by exhausting all the possible 2^25= 33554432 models. It seems fine to me if you want to continue using LASSO.

Regarding the nested versus non nested: this defines how you select the complexity of your model and how you report the error of your model. This holds regardless of using the feature selection method (exhaustive search, greedy search, LASSO,..).

First of all you need to report the error on a part of data that the model is not fitted to it. In technical terms you need separate train and test sets. Since your data set is small (only 90 samples), it is not very wise to put aside a part of it for testing. Hence you need to use cross-validation to measure the performance of your model. For example you break your data into 4 sets. Then train on 3 sets and test on the set left out of training. Rotate the set that is left out of training and to this 4 times and average the error.

A part of model fitting on the 3 sets in the above example is determining their complexity (in case of exhaustive selection is the input features and in case of LASSO is the amount of regularization). You may first determine the complexity that minimizes the CV error and then use that complexity to train all the 4 models. In this way however all the 4 models share the same complexity parameter that is optimized for the whole data set. Hence you will get an overly optimistic error. Instead in each CV fold you can find the complexity parameter based on the data in that fold. This is where nested CV comes.

So the process of training the model in a fold containing (s1,s2,s3) for training and s4 for testing will be:

1. Determine the complexity of model via cross-validation on (s1,s2,s3).(nested part of CV)
2. Fit the model on (s1,s2,s3) using the complexity parameter obtained in step 1.
3. Measure your performance on s4 Repeat the process for all folds.

BIC/AIC/ CV/ LARS CV : These are the methods that the amount of regularization is determined. The difference between CV and LARS CV should be small. In BIC and AIC terms are added to the cost function that penalize the complexity and as far as I know they used less commonly.

• "Determine the complexity of model via cross-validation on (s1,s2,s3).(nested part of CV)" is the nested cv a 3-fold cause the outer cv is 4-fold if I understood correctly. – Da Kor Mar 29 '17 at 15:53
• Yes. That is correct. – Hooman Mar 29 '17 at 15:59
• When you say "In your case you can easily search for the best feature set by exhausting all the possible 2^25= 33554432 models." is there a way to do that in Python or is it only an R thing? Is it the same as the best subsets regression procedure? – Da Kor Mar 29 '17 at 16:24
• I have another question. If I do a n-fold inner, and k-fold outer can I use the features that are present in every model and run a multiple linear regression (eve though bias as I'll retrain to the same data)? – Da Kor Mar 29 '17 at 19:07
• If your goal is to find the smallest features that needs to be used for prediction then choosing the features that show up in all models seems like a logical solution. Alternatively you can choose the largest L1 regularization parameter from all the models and fit a model using this this regularization constant. – Hooman Mar 29 '17 at 21:38