Is there any possibility of overfitting even after higher AUC, specificity and sensitivity obtained through repeated k-fold cross-validation?

Question

I have built a model where a 10-fold cross-validation was performed 10 times. The average AUC, MCC, specificity, sensitivity of 10 times were reported as the prediction performance. Yet some people are saying that my model can be overfitted even after adopting the cross-validation. Is there really any possibility of overfitting?

Dataset details:

Positive instances: 182

negative instances: 1637

classifier: Biased SVM

Average performances of 10 repetitions of 10-fold cv:

AUC: 99.49(%), MCC: 97.64(%), Sn: 99.40(%), Sp: 98.58(%)

Answer 1

Yes, for example if you include the target variable (or a measure very similar to it) among the features it can happen that you have a very high AUC but once you'll try to make a prediction your model will not perform well.

Check which are the most important features.

Answer 2

Another very frequent reason why CV-estimates become overoptimistic for generalization error is structure in the underlying data, e.g. clusters of cases or hierarchy (1:n relations).

If you have clusters but split by case rather than by cluster, you're likely to test cases whose cluster-neighbors were in the training set. Performance for such cases may be better than for cases from new clusters.

---

And, of course, if the results of the cross validation that you use to estimate generalization error were used to steer any modeling decisions, that's another possible source of overfitting.