0
$\begingroup$

I have a dataset of 150 patients (2:1 ratio of classes) and 78 features. I performed backwards elimination using logistic regression feature importance to end up with 13 features (SVC classifier). I used nested cross validation for it and for the hyperparams. Then I calculate the leave-one-out AUC on the whole dataset as my last step and I get 0.94 AUC. I believe everything is correct BUT I performed an analysis to see how much dropping X patients affect the AUC and I get the attached plot (I drop X% on each class). My question is: am I overfitting somehow or it is expected to see such a 'stable' AUC because the model works?

Another comment: Even if I drop 90% of the data (stratified) I still get an average AUC of 0.8

Thanks

enter image description here

Improve this question
$\endgroup$
1
$\begingroup$

Over-fitting, most probably. Your model might fit this particular set of data very well, but it could be unlikely to generalize well to another sample of data from the same population.

With 150 patients total and a 2:1 class ratio, you only have about 50 in the minority class. With biomedical data such as you have, an unpenalized model is likely to be overfit if you are evaluating more than 4 or so predictors in your model. (The usual rule of thumb is 10-20 members of the minority class per evaluated predictor.)

Simply testing by leave-one-out cross-validation (CV) on the full data set it not the best way to evaluate your model-building procedure. Leave-one-out CV is prone to the high variance typical of an over-fit model when applied to new samples from the population.

For validation of a modeling procedure, it's more reliable to develop models with the same procedure (including all of the predictor-selection and hyperparameter-selection steps) on multiple (tens to hundreds) of bootstrap samples from the data, and then testing performance on the full original data set. My suspicion is that you will find less-than-ideal performance with your backward elimination approach. Backward elimination might be less objectionable than forward-selection methods, but it still runs the risks of all automated model selection methods.

Improve this answer
$\endgroup$
6
  • $\begingroup$ What you are suggesting is training on the bootstrap samples (with replacement I assume) and test on the full original data? or test on the remaining data (i.e. repeated train/test split but with bootstrap)? $\endgroup$ – Luis Pinto Jan 25 '20 at 21:49
  • 1
    $\begingroup$ @LuisPinto yes, sampling with replacement to get multiple training sets of the original size, then testing on the full original data. Each time repeat the entire model-building process (hyperparameter and predictor selection). Sampling with replacement from the original data sample represents something like taking the original data sample from the underlying population. So the performances of the models built from bootstrapped samples, tested on the full original data sample, provides an estimate of how the model built from the full original data sample would work on the underlying population. $\endgroup$ – EdM Jan 25 '20 at 22:00
  • $\begingroup$ And how do I choose the features from that pool of features? The ones that were repeated the most? $\endgroup$ – Luis Pinto Jan 25 '20 at 22:42
  • $\begingroup$ @LuisPinto the recommended steps evaluate the model-building approach as applied to your data. So if the performance of the backward-elimination approach on the bootstrap samples, each evaluated on the full original data set, is OK, then you just use the original model derived from the full data set. My fear is that the performance of the models obtained via backward-elimination won't be OK. In that case you would be better off with a penalized approach like LASSO, ridge regression, or elastic net to avoid overfitting and get a model that is likely to generalize better to new data samples. $\endgroup$ – EdM Jan 25 '20 at 23:26
  • 1
    $\begingroup$ @LuisPinto I'm not familiar enough with scikit-learn to know. Ridge regression places L2 penalization on regression coefficients, but my sense is that SVC tends to draw explicit yes/no boundaries, emphasizing data near the boundary. I'm not sure that would give the same results as a logistic ridge regression, which would model log-odds and thus probabilities of class membership based on all data points. It's generally better to get a good probability model first and then set a classification boundary, if needed, based on your cost tradeoff between false-positive and false-negative results. $\endgroup$ – EdM Jan 27 '20 at 0:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.