I am dealing with an imbalanced classification problem and used oversampling on my training set to to predict on my testing set.

My PI insists on presenting evaluation metrics of the different trained models based on training set predictions, in addition to their testing set predictions. This is a clinical study, where usually logistic regression is conventionally used to predict an outcome and then use it on a separate set of patients (testing set) to "externally" validate it. In our case, we used other ML algorithms in addition to conventional STATA-like LR to show that they perform better.

2 questions:

  1. Is it conventionally acceptable to present such metrics (recall, precision, F1, AUC, etc) by predicting on the training data, in addition to the ones we get from predicting on testing data?
  2. If yes, shall I run such predictions on the oversampled training set (where the training actually happened) or on the non-oversampled training set?

Thank you

  1. No, this is not common. Presenting results on training data amounts to showing how well your algorithms fit (not predict!) the data. But we presumably care about performance on new unseen data - and in-sample fit is a notoriously poor indicator of out-of-sample performance, because it is highly susceptible to overfitting.

    Bottom line: presenting in-sample results adds no information, only noise. If you can convince your PI not to show this, you will have done science a service.

  2. Per above, don't.

  3. Unbalanced data are usually no problem, and oversampling will not solve a non-problem.

  • $\begingroup$ Thank you, that’s what I suspected. I only planned to included training and testing AUCs as surrogate markers to overfitting (ie statistically nonsignificant difference in AUCs -> less likely to have overfitting). Of note, oversampling was very useful for my model where i had to capture 3% minority class with good sensitivity (mortality), with good performance on testing set (non-oversampled). Training on non-oversampled data was efficient only for detecting majority class , not useful in my case. $\endgroup$ – Paris Char Mar 5 at 3:05
  • $\begingroup$ Note that sensitivity suffers from the same problems as accuracy. I always recommend aiming rather for well calibrated probabilistic predictions. Good luck! $\endgroup$ – Stephan Kolassa Mar 5 at 7:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.