1
$\begingroup$

It makes sense to me that we can use the ROC-AUC and PR-AP scores of the validation sets during CV to tune our model hyperparameter selection. And when reporting the models final performance, it makes sense to create a PR or ROC curve on the test data set to illustrate performance on unseen data. But in terms of selecting a decision threshold for the model probabilities, does it make the most sense to create the PR/ROC curve(s) from validation data during CV, or from test data after the final model is already tuned via CV? We generally try to avoid that the test data affects how the model is created/tuned, but the decision boundary is not technically part of the model's hyper-parameters (as elaborated on in Is decision threshold a hyperparameter in logistic regression?), so is it fine in this case and this would not count as test data leakage?

This question is similar to threshold choice for binary classifier: on training, validation or test set?, but specifically about creating the PR/AUC curves where the suggested approach in that answer would not lead to the creation of a PR/AUC curve. It is also similar to the question Why ROC Curve on test set?, but that does not explicitly address why this is not considered test data leakage (which would lead to an overly optimistic evaluation)

$\endgroup$

1 Answer 1

2
$\begingroup$

Use training data to choose the threshold.

selecting a decision threshold ... from test data ?

No.

We train a model to learn the shape of the data. We choose a classification decision threshold to express how we value precision versus recall, and that threshold plays a large role in measured performance of the business use case once the model is deployed. It is a model parameter.

We generally try to avoid that the test data affects how the model is created / tuned ...

That's right.

We're training a classifier that should generalize, that should perform well on tomorrow's unseen data, and the next day after that. Using test data to choose the parameter is cheating. We use test data to evaluate the trained model, full stop.

Let me quote from Abu-Mostafa, et al., "Learning From Data", §5.3:

Data snooping is the most common trap for practitioners in learning from data. The principle involved is simple enough,

If a data set has affected any step in the learning process, its ability to assess the outcome has been compromised.

Applying this principle, if you want an unbiased assessment of your learning performance, you should keep a test set in a vault and never use it for learning in any way.

Or, if you do peek into the vault, there should be another vault available with additional unseen data for next week's tuning. Or you should be able to collect fresh data, perhaps via daily collection of transaction logs.

$\endgroup$
3
  • $\begingroup$ Thank you for the reply! It makes sense to me to not look at the test data at all, not even for the decision boundary. One thing that has me confused with the approach of using validation data, is if we create one PR/ROC curve per CV fold and then average them, or concatenate all the predictions and then create a single PR/ROC curve to choose the threshold. Since every fold can be thought of having its own model, I'm not sure about the best way of aggregating them (although the hyperparams are the same per fold, the regular params vary as the model learns from the data) $\endgroup$ Nov 19, 2023 at 22:37
  • $\begingroup$ This post and the linked paper in one of the answers argues that is shouldn't matter much and that if anything the concatenated CV set is better. So I guess something like this example in sklearn, but for the PR/ROC curve, would you agree with that? $\endgroup$ Nov 19, 2023 at 22:39
  • $\begingroup$ I agree with that. $\endgroup$
    – J_H
    Nov 19, 2023 at 23:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.