I am working on building a predictive model on medical images based on global images features extraction (texture).
As I only have 51 patients, I want to build a simple model with cross-validation. The outcome is binary, so first I think the most appropriated strategy is to use logistic regression. My current strategy is to use forward selection based on cross-validation AUC for a LASSO regularized logistic regression model. To avoid potential over-fitting caused by forward selection, I iteratively select the features according to the mean CV AUC:
rcv = RepeatedStratifiedKFold(n_repeats=100, n_splits=5, random_state=1) #setting the repeated cross validation strategy : 100 times 5 folds CV with shuffling at each of the 100 times max_number_of_features = 23 #number of features in the dataset after a first unsupervised feature selection (really need to avoid redundancies) mean_cv_auc_over_i =  #list which will contain the mean CV AUC by number of features selected std_cv_auc_over_i =  #associated std for i in range(1, max_number_of_features+1): feature_set =  for num_features in range(i): mean_auc_list =  std_auc_list =  steps = list() steps.append(('scaler', StandardScaler())) #scaling of the features using zscore : feature = (feature-mean(features))/std(features) steps.append(('model', LogisticRegression(penalty='l1', solver='liblinear', class_weight='balanced', C=4))) #LASSO regularized logistic regression with C=4 based on grid search with all 23 features (done before...) pipeline = Pipeline(steps=steps) for feature in Xdf.columns: if feature not in feature_set: f_set = feature_set.copy() f_set.append(feature) scores = cross_val_score(pipeline, Xdf[f_set], ydf, scoring='roc_auc', cv=rcv) #compute all 500 AUC according to the cross validation strategy mean_auc = scores.mean() #compute the mean AUC std_auc = scores.std() #compute the associated std mean_auc_list.append((mean_auc, feature)) std_auc_list.append((feature, std_auc)) mean_auc_list.sort(key=lambda x : x, reverse = True) #sort mean AUCs and associated features std_auc_list = dict(std_auc_list) std_auc_list = [(std_auc_list[key], key) for key in [x for x in mean_auc_list]] #associated std feature_set.append(mean_auc_list) #append the selected new feature to the ordered set mean_cv_auc_over_i.append((i, mean_auc_list)) #append the mean CV AUC for this number of selected feature std_cv_auc_over_i.append((i, std_auc_list)) #append the associated std
I need help with 3 things:
- Is my strategy of forward selecting the features according to the mean CV AUC OK here?
- Is it OK to keep the LASSO regularization (which also selects features), in the loop?
- (If yes for the second question) As you can see, I fixed the LASSO parameter C to 4 according to a grid search done before this process, with all 23 features (which doesn't really seem appropriate)... Do you have any idea how to optimize the LASSO's C parameter inside this process instead of before with all features?
The second big problem is that logistic regression suffers from a particularly bad type of omitted-variable bias. In logistic regression, if you omit a predictor that's associated with outcome you can tend to bias the magnitudes of coefficients of included predictors down toward 0 even for predictors not correlated with the omitted predictor. So any one-feature-at-a-time approach for logistic regression has substantial disadvantages (even putting aside the low sensitivity of AUC for distinguishing among models). That provides an answer ("No") to your point 1.
I know a little about omitted variable bias, but I am not sure on how to consider this in case of image (or signal) texture analysis. Indeed, the textural features that we extract in the images are well defined mathematically, but their signification in term of biology is not defined. To understand why they permit classification and what they "show" in term of biology is part of the work. Also, all theses textural features can be highly correlated between each other. That is why I first use an unsupervised iterative feature elimination using correlations or multiple correlations. A point I do not understand here is why an omission of a variable can end up in decreasing the magnitude of coefficient of others in my case ? I understand why it can increase it, but not decrease, because of the fact that these features are signal features, and so are always correlated between each other and with the dependent variable in directions where the coefficient magnitudes will never be decreased.
For example : x1 and x2 are negatively correlated. In my case, if x1 is positively correlated with y the dependent variable, x2 will be negatively correlated with y systematically. So, if we omit x2, the coefficient of x1 will be more positive (upward biased), and on the other hand if we omit x1, the coefficient of x2 will be more negative (downward biased), increasing the magnitude in both cases. I am not sure but I suppose this is true because in the case of textural features, correlations can be seen as redundancies. What do you think about this ?
One big problem here is the combination of LASSO with a small number of cases. LASSO at its optimum penalty factor will typically retain a number of predictors close to the number that can be comfortably be assessed without overfitting. In a logistic regression, that's about 1 predictor for every 15 or so members of the minority class. With only 51 cases total there can be no more 25 members of the minority class, so LASSO will probably only return about 2 or 3 of your features with non-zero coefficients.
About the choice of LASSO, I chose this approach because this was the one giving me the best mean AUC with the smallest variance. Why do you think AUC is not an good choice of criterion compared to deviance and for assessing the probability modelling ? Thank you.