Are repeated cross validation, forward feature selection, and LASSO compatible?

Question

I am working on building a predictive model on medical images based on global images features extraction (texture).

As I only have 51 patients, I want to build a simple model with cross-validation. The outcome is binary, so first I think the most appropriated strategy is to use logistic regression. My current strategy is to use forward selection based on cross-validation AUC for a LASSO regularized logistic regression model. To avoid potential over-fitting caused by forward selection, I iteratively select the features according to the mean CV AUC:

rcv = RepeatedStratifiedKFold(n_repeats=100, n_splits=5, random_state=1)    #setting the repeated cross validation strategy : 100 times 5 folds CV with shuffling at each of the 100 times

max_number_of_features = 23    #number of features in the dataset after a first unsupervised feature selection (really need to avoid redundancies)

mean_cv_auc_over_i = []    #list which will contain the mean CV AUC by number of features selected 
std_cv_auc_over_i = []    #associated std

for i in range(1, max_number_of_features+1):
    feature_set = []

    for num_features in range(i):
        mean_auc_list = []
        std_auc_list = []

        steps = list()
        steps.append(('scaler', StandardScaler()))    #scaling of the features using zscore : feature = (feature-mean(features))/std(features)
        steps.append(('model', LogisticRegression(penalty='l1', solver='liblinear', class_weight='balanced', C=4)))    #LASSO regularized logistic regression with C=4 based on grid search with all 23 features (done before...)
        pipeline = Pipeline(steps=steps)

        for feature in Xdf.columns:
            if feature not in feature_set:
                f_set = feature_set.copy()
                f_set.append(feature)

                scores = cross_val_score(pipeline, Xdf[f_set], ydf, scoring='roc_auc', cv=rcv)    #compute all 500 AUC according to the cross validation strategy
                mean_auc = scores.mean()    #compute the mean AUC
                std_auc = scores.std()    #compute the associated std

                mean_auc_list.append((mean_auc, feature))
                std_auc_list.append((feature, std_auc))

        mean_auc_list.sort(key=lambda x : x[0], reverse = True)    #sort mean AUCs and associated features
        std_auc_list = dict(std_auc_list)
        std_auc_list = [(std_auc_list[key], key) for key in [x[1] for x in mean_auc_list]]    #associated std
        feature_set.append(mean_auc_list[0][1])    #append the selected new feature to the ordered set

    mean_cv_auc_over_i.append((i, mean_auc_list[0][0]))    #append the mean CV AUC for this number of selected feature
    std_cv_auc_over_i.append((i, std_auc_list[0][0]))    #append the associated std

I need help with 3 things:

1. Is my strategy of forward selecting the features according to the mean CV AUC OK here?
2. Is it OK to keep the LASSO regularization (which also selects features), in the loop?
3. (If yes for the second question) As you can see, I fixed the LASSO parameter C to 4 according to a grid search done before this process, with all 23 features (which doesn't really seem appropriate)... Do you have any idea how to optimize the LASSO's C parameter inside this process instead of before with all features?

---

edit:

The second big problem is that logistic regression suffers from a particularly bad type of omitted-variable bias. In logistic regression, if you omit a predictor that's associated with outcome you can tend to bias the magnitudes of coefficients of included predictors down toward 0 even for predictors not correlated with the omitted predictor. So any one-feature-at-a-time approach for logistic regression has substantial disadvantages (even putting aside the low sensitivity of AUC for distinguishing among models). That provides an answer ("No") to your point 1.

I know a little about omitted variable bias, but I am not sure on how to consider this in case of image (or signal) texture analysis. Indeed, the textural features that we extract in the images are well defined mathematically, but their signification in term of biology is not defined. To understand why they permit classification and what they "show" in term of biology is part of the work. Also, all theses textural features can be highly correlated between each other. That is why I first use an unsupervised iterative feature elimination using correlations or multiple correlations. A point I do not understand here is why an omission of a variable can end up in decreasing the magnitude of coefficient of others in my case ? I understand why it can increase it, but not decrease, because of the fact that these features are signal features, and so are always correlated between each other and with the dependent variable in directions where the coefficient magnitudes will never be decreased.

For example : x1 and x2 are negatively correlated. In my case, if x1 is positively correlated with y the dependent variable, x2 will be negatively correlated with y systematically. So, if we omit x2, the coefficient of x1 will be more positive (upward biased), and on the other hand if we omit x1, the coefficient of x2 will be more negative (downward biased), increasing the magnitude in both cases. I am not sure but I suppose this is true because in the case of textural features, correlations can be seen as redundancies. What do you think about this ?

One big problem here is the combination of LASSO with a small number of cases. LASSO at its optimum penalty factor will typically retain a number of predictors close to the number that can be comfortably be assessed without overfitting. In a logistic regression, that's about 1 predictor for every 15 or so members of the minority class. With only 51 cases total there can be no more 25 members of the minority class, so LASSO will probably only return about 2 or 3 of your features with non-zero coefficients.

About the choice of LASSO, I chose this approach because this was the one giving me the best mean AUC with the smallest variance. Why do you think AUC is not an good choice of criterion compared to deviance and for assessing the probability modelling ? Thank you.

Answer 1

Is my strategy of forward selecting the features according to the mean CV AUC OK here?

Since you are using Python you should be aware of Recursive Feature Selection CV which in essence is doing what you are, but this gives you the advantage of working with pipelines

Is it OK to keep the LASSO regularization (which also select feature), in the loop?

This is ok since with the feature selection you are pointing to the combination of features that "optimize" the metric, whereas L1 regularization is punishing the non-significant features for the specific model

If yes for the second question, as you see, I fixed the LASSO parameter C to 4 according to a grid search done before this process, with all 23 features (which sounds not really appropriated)... Have you an idea on how to optimize the C parameter of the LASSO inside this process instead of before with all features?

You can use LogisticRegressionCV which will optimize the regularization parameter via cross-validation for you at each fit stage.

This is a working example using a dummy dataset of how this should be used:

from sklearn.linear_model import LogisticRegressionCV
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.feature_selection import RFECV
from sklearn.model_selection import train_test_split

from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y= True)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .2, random_state = 42)

scoring = "roc_auc"

model = Pipeline([("scaler", StandardScaler()), ("model",RFECV(estimator = LogisticRegressionCV(max_iter = 10000,class_weight="balanced", scoring= scoring, random_state= 42)                                                           ,scoring = scoring))]).fit(X_train, y_train)

model.score(X_test, y_test)

Note that as per 1st question both, the feature selection and C parameter are to optimize roc auc. You can test by yourself that when modifying this, i.e changing the metric on any of the two processes, the scoring will decrease (ej. optimizing the metric for feature selection as accuracy and roc auc for C in logistic regression model)

Answer 2

One big problem here is the combination of LASSO with a small number of cases. LASSO at its optimum penalty factor will typically retain a number of predictors close to the number that can be comfortably be assessed without overfitting. In a logistic regression, that's about 1 predictor for every 15 or so members of the minority class. With only 51 cases total there can be no more 25 members of the minority class, so LASSO will probably only return about 2 or 3 of your features with non-zero coefficients.

The second big problem is that logistic regression suffers from a particularly bad type of omitted-variable bias. In logistic regression, if you omit a predictor that's associated with outcome you can tend to bias the magnitudes of coefficients of included predictors down toward 0 even for predictors not correlated with the omitted predictor. So any one-feature-at-a-time approach for logistic regression has substantial disadvantages (even putting aside the low sensitivity of AUC for distinguishing among models). That provides an answer ("No") to your point 1.

It sounds like you might be better off using a method that uses all of your features in some way, like ridge regression. An Introduction to Statistical Learning shows, in Section 6.6, how to use cross-validation directly to choose an optimal penalty value for ridge regression.* For a logistic regression you would be best off using the deviance criterion (or the equivalent log-loss criterion) for optimizing the penalty level via cross-validation rather than AUC. You might also consider a different approach to modeling class probabilities that can use all of your features while avoiding overfitting, say one that learns slowly like boosted regression trees.

---

*or for LASSO if your still want to use that, providing answers to your points 2 and 3: you can and should choose the penalization factor through cross validation by optimizing an appropriate measure of model performance.