# On which set (train/val/test) do people calculate F1 score, precision and recall?

This may be a stupid question, but when I was looking at the definition of precision/recall etc. it was not mentioned anywhere which set (training/validation/test) this metric should be calculated against.

Is this metric used to select the best model or evaluate the final model?

• The score on training data is the least important. After that, you would calculate it on a validation set to tune hyperparameters and then on a final test set to estimate model performance. However, keep in mind that $F_1$, like accuracy, sensitivity, and specificity, is an improper scoring rule that does not consider the predicted probabilities.
– Dave
Jun 17, 2021 at 17:33
• Dave said it well, and it deserves to be repeated: the test set is never for model selection. Don't look at it until you've done all the tuning and engineering you want, and you're done re-trianing/tweaking your model. It is your proxy for the real world, and in the real world you don't get do-overs. Jun 17, 2021 at 18:02