I am dealing with an imbalanced classification problem and used oversampling on my training set to to predict on my testing set.
My PI insists on presenting evaluation metrics of the different trained models based on training set predictions, in addition to their testing set predictions. This is a clinical study, where usually logistic regression is conventionally used to predict an outcome and then use it on a separate set of patients (testing set) to "externally" validate it. In our case, we used other ML algorithms in addition to conventional STATA-like LR to show that they perform better.
- Is it conventionally acceptable to present such metrics (recall, precision, F1, AUC, etc) by predicting on the training data, in addition to the ones we get from predicting on testing data?
- If yes, shall I run such predictions on the oversampled training set (where the training actually happened) or on the non-oversampled training set?