Nested cross-validation or bootstrapping and non-nested cross validation?

Question

I have 25 correlated independent variables and one dependent variable that is an aggregated score of a Likert scale. I also have 90 samples. I want to do variable selection for linear regression so I am using LASSO. In python I used LASSO CV (coordinate descent). Q1 I can use LASSO AIC/BIC, LASSO CV, LASSO LARS CV is there any reason to pick one over the other? Q2 I tried LASSO CV with nested cross validation as I was told that it will calculate better my hyper parameters since I have only 90 samples but the (3-fold inner/3-fold outer) but the 3 models I get are very different to each other. Do I need to do bootstrapping and do non nested instead?

Answer 1

Hi this should probably be a comment and not an answer, but I have written a small R code (see here) that can perform LASSO with nested-CV, in which the inner loop performed LOOCV, which can help make the output models more consistent (so n-fold inner, and k-fold outer). I hope this helps.

Answer 2

Firstly if your models in each CV folds are too different then this is a sign of unstability of model fitting. Either your model is too complex for data or the features are too correlated. You may use a combination of norm 1 and norm 2 regularization to improve the stability of LASSO (known as elastic net regularization https://en.wikipedia.org/wiki/Elastic_net_regularization)

Secondly with your data set you may not really need LASSO for feature selection. LASSO or norm 1 regularization is especially useful when the data set is large. In your case you can easily search for the best feature set by exhausting all the possible 2^25= 33554432 models. It seems fine to me if you want to continue using LASSO.

Regarding the nested versus non nested: this defines how you select the complexity of your model and how you report the error of your model. This holds regardless of using the feature selection method (exhaustive search, greedy search, LASSO,..).

First of all you need to report the error on a part of data that the model is not fitted to it. In technical terms you need separate train and test sets. Since your data set is small (only 90 samples), it is not very wise to put aside a part of it for testing. Hence you need to use cross-validation to measure the performance of your model. For example you break your data into 4 sets. Then train on 3 sets and test on the set left out of training. Rotate the set that is left out of training and to this 4 times and average the error.

A part of model fitting on the 3 sets in the above example is determining their complexity (in case of exhaustive selection is the input features and in case of LASSO is the amount of regularization). You may first determine the complexity that minimizes the CV error and then use that complexity to train all the 4 models. In this way however all the 4 models share the same complexity parameter that is optimized for the whole data set. Hence you will get an overly optimistic error. Instead in each CV fold you can find the complexity parameter based on the data in that fold. This is where nested CV comes.

So the process of training the model in a fold containing (s1,s2,s3) for training and s4 for testing will be:

1. Determine the complexity of model via cross-validation on (s1,s2,s3).(nested part of CV)
2. Fit the model on (s1,s2,s3) using the complexity parameter obtained in step 1.
3. Measure your performance on s4 Repeat the process for all folds.

BIC/AIC/ CV/ LARS CV : These are the methods that the amount of regularization is determined. The difference between CV and LARS CV should be small. In BIC and AIC terms are added to the cost function that penalize the complexity and as far as I know they used less commonly.