Shall I present predictions on the (oversampled) training set as well?

Question

I am dealing with an imbalanced classification problem and used oversampling on my training set to to predict on my testing set.

My PI insists on presenting evaluation metrics of the different trained models based on training set predictions, in addition to their testing set predictions. This is a clinical study, where usually logistic regression is conventionally used to predict an outcome and then use it on a separate set of patients (testing set) to "externally" validate it. In our case, we used other ML algorithms in addition to conventional STATA-like LR to show that they perform better.

2 questions:

1. Is it conventionally acceptable to present such metrics (recall, precision, F1, AUC, etc) by predicting on the training data, in addition to the ones we get from predicting on testing data?
2. If yes, shall I run such predictions on the oversampled training set (where the training actually happened) or on the non-oversampled training set?

Thank you

Answer 1

1. No, this is not common. Presenting results on training data amounts to showing how well your algorithms fit (not predict!) the data. But we presumably care about performance on new unseen data - and in-sample fit is a notoriously poor indicator of out-of-sample performance, because it is highly susceptible to overfitting.

   Bottom line: presenting in-sample results adds no information, only noise. If you can convince your PI not to show this, you will have done science a service.

2. Per above, don't.

3. Unbalanced data are usually no problem, and oversampling will not solve a non-problem.