Are evaluation metrics computed on training dataset?

Question

Based on my own studies and questions on this site, my understanding is that evaluation metrics (accuracy, precision, recall) are only calculated on the test dataset.

The training dataset is used to build the model. The test dataset is used to evaluate the model by having actual labels and then predicted labels using the built model (a confusion matrix that can be used to calculate evaluation metrics).

So, am I correct to say evaluation metrics are only calculated on the test dataset? Or am I misunderstanding something and evaluation metrics are computed on both the training dataset and test dataset?

Answer 1

It depends on what you mean by "evaluation metrics". If you mean "the metrics calculated to evaluate the model", then those are the metrics calculated on the "evaluation set", i.e. the test set. If you mean "the metrics that can be used to evaluate the performance of the model", then this is just a list of metrics used for this purpose. In general, you usually calculate the metrics on the training set and on the test set (and sometimes also on the validation/dev set). The metrics calculated on the training set tell you what is the in-sample performance of the model, the metrics calculated on the test set tell you about the out-of-sample performance. The metrics calculated on the training set tell you how does the model performs on the "seen" data, if the performance is poor, your model is underfitting. If the model performs well on the training set, but poorly on the test set, it tells you that the model is overfitting to the training data, so we usually need to calculate the metrics on the both sets.

Answer 2

You are correct. You are trying to evaluate the performance of the model, so you look at what the model implied results(i.e. computing accuracy, recall, etc.). You treat the training data as true and so it wouldn't make sense to evaluate this data set when you want to know how your model preforms in test data.

There are times when you do compute these values on the training data. For example when you do model selection. So in this case you might have several possible choices of model and you compute something like accuracy to choose the best model to then apply to the test data.