How to present ML model performance before deployment? I have machine learning model performance metrics on training set with cross validation and metrics from when I ran the model on the test set. The model performs similar on both sets, so I want to deploy model and make predictions.
Which performance metrics are the ones that matter more and that others care about, training set or test set metrics?  Which results get displayed more prominently? From research papers, it’s not clear to me which dataset is being used to show, for example AUC with confidence intervals.
My question: When most people talk about how their model performs, are they usually talking about training set performance or test set performance?
 A: Training set performance is all but irrelevant.$^{\dagger}$ You’d be interested in how the model performs on data to which it has not been exposed, since that’s how it will be used once it is deployed. For example, you don’t care about predicting yesterday’s stock prices; you care about tomorrow’s stock prices.
$^{\dagger}$I could see looking at it to check it your model is overfit or underfit. If both in-sample and out-of-sample performance are poor, you have not captured the predictive patterns. Maybe you’re missing a variable, for instance. If your in-sample performance is stellar but out-of-sample performance is poor, perhaps you’ve overfit to the in-sample data.
A: Your last question - it's only test set performance that matters. So, after training has ended and model selection taken place, you can compute metrics like AUC (for the ROC-curve) based on your test set. There is precision and recall - in a medical setting you will compute predictive value positive and predictive value negative, sensitivity and specificity.
However, when your classifier contains more than two classes, several other metrics can be computed than just the (conditional) correctness / error rates.
For an overview of generic quality measures (beyond neural networks) see for example:
M. Egmont-Petersen, J.L. Talmon, J. Brender, P. NcNair. "On the quality of neural net classifiers," Artificial Intelligence in Medicine, Vol. 6, No. 5, pp. 359-381, 1994.
