1
$\begingroup$

This may be a stupid question, but when I was looking at the definition of precision/recall etc. it was not mentioned anywhere which set (training/validation/test) this metric should be calculated against.

https://developers.google.com/machine-learning/crash-course/classification/precision-and-recall

Is this metric used to select the best model or evaluate the final model?

Can someone please elaborate?

Improve this question
$\endgroup$
2
  • 3
    $\begingroup$ The score on training data is the least important. After that, you would calculate it on a validation set to tune hyperparameters and then on a final test set to estimate model performance. However, keep in mind that $F_1$, like accuracy, sensitivity, and specificity, is an improper scoring rule that does not consider the predicted probabilities. $\endgroup$
    – Dave
    Jun 17, 2021 at 17:33
  • 3
    $\begingroup$ Dave said it well, and it deserves to be repeated: the test set is never for model selection. Don't look at it until you've done all the tuning and engineering you want, and you're done re-trianing/tweaking your model. It is your proxy for the real world, and in the real world you don't get do-overs. $\endgroup$
    – Arya McCarthy
    Jun 17, 2021 at 18:02

1 Answer 1

Reset to default
2
$\begingroup$

Understanding this distinction is important in ML.

Usually, you use precision, recall, and F1 to evaluate the (generalization) performance of your model.

Therefore, you compute these on the test set.

Separately from this, you also need to select a single metric to optimize your model hyperparameters during training and validation, eg, via (inner) cross-validation. Often this is accuracy or ROC AUC but it could be anything. This is where the art comes in. You may stick to a default metric to begin with.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.