1
$\begingroup$

I have built a model where a 10-fold cross-validation was performed 10 times. The average AUC, MCC, specificity, sensitivity of 10 times were reported as the prediction performance. Yet some people are saying that my model can be overfitted even after adopting the cross-validation. Is there really any possibility of overfitting?

Dataset details:

Positive instances: 182

negative instances: 1637

classifier: Biased SVM

Average performances of 10 repetitions of 10-fold cv:

AUC: 99.49(%), MCC: 97.64(%), Sn: 99.40(%), Sp: 98.58(%)

$\endgroup$
2
$\begingroup$

Yes, for example if you include the target variable (or a measure very similar to it) among the features it can happen that you have a very high AUC but once you'll try to make a prediction your model will not perform well.

Check which are the most important features.

$\endgroup$
1
$\begingroup$

Another very frequent reason why CV-estimates become overoptimistic for generalization error is structure in the underlying data, e.g. clusters of cases or hierarchy (1:n relations).

If you have clusters but split by case rather than by cluster, you're likely to test cases whose cluster-neighbors were in the training set. Performance for such cases may be better than for cases from new clusters.


And, of course, if the results of the cross validation that you use to estimate generalization error were used to steer any modeling decisions, that's another possible source of overfitting.

$\endgroup$
2
  • $\begingroup$ so what's the solution then? The 10-fold cross-validation was performed 10 times for reducing the bias. Each time train and test set were picked randomly. After doing so many things, how cluster neighbors can be handled? Stratified k-fold? $\endgroup$ – Sabit Ahmed Jun 8 '20 at 4:23
  • $\begingroup$ Again, the AUC, sensitivity and specificity is higher and almost identical to each other. For the case of accuracy, it does not work for overfitting. But all possible metrics are giving higher result. What about that? @cbeleites $\endgroup$ – Sabit Ahmed Jun 8 '20 at 4:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.