Classification Threshold Optimization after GridSearchCV

Question

In my machine learning problem I am using a CNN to classify images.

Since my dataset is imbalanced I want to perform classification probability threshold tuning so I can find the optimal balance between sensitivity and specificity using G-mean over the returned values of the ROC-curves (explanation link).

I have split the dataset into train-test. I want to do Cross Validation on the training set only so I can hypertune the network parameters (mainly batch size, number of epochs and optimizer). For finding the best model I am thinking of using the roc_auc_score since its value is not dependent on the classification threshold applied.

I think that is probably a bad practice to use the test set to do this threshold determination directly (although I have seen it done).

My question comes down to this: After finding the best model how can I find the optimal threshold value? I am thinking of the following possibilities:

• A: Assume that the best set of model/parameters doesn't overfit on the training data (since it's the best performing model on the gridsearch) and retrain on the full training set, obtain the classification probabilities for the training set, get optimal threshold.
• B: Instead of dividing the original dataset into train-test, create a third set (so train-validation-test split). Find the best model on training set (through gridSearchCV), retrain on the training set and predict the validation set, optimize the threshold on this prediction.
• C: Use part of the training data to train( like 70%), another part to validate. Find the optimal threshold.
• D: With the best model perform CV again. Log the CV results for each fold. For a set of threshold let's say [0.1, 0.2, ..., 0.9] check which one across the folds has better average score (for example f1-macro). On all the possibilities, I would then use the full training data (+ validation on the case of B and C) to train the model, get the test classification probabilities, use the previously determined threshold to the prediction classes, compute classification metrics.

I think that in A, if my best model overfits, the threshold will be wrong (although I can always compare it to the 0.5 threshold and see if it improves or not).

In B, I am giving up samples that could be helpful on the gridsearchCV.

In C, like in A I am also retraining on data that was use for model selection. The difference to A is that the prediction is not done on data that has been fit.

I don't really see any disadvantage in D. Although I didn't find that suggestion anywhere, which I find a little weird. I guess I don't really need to perform CV again if I just log the best threshold for all the gridsearched models.

What do you think is the best approach? Is there a better way to do this?

Answer 1

I would treat the threshold as one of the hyper-parameters tuned during the grid-search procedure. In this case you are not performing cross-validation for performance evaluation, just for hyper-parameter tuning so bias is not too relevant. That way you can tune the hyper-parameters to optimise the criterion that is more directly relevant to the application.

I think that is probably a bad practice to use the test set to do this threshold determination directly (although I have seen it done).

Absolutely, the test set should be used only for estimating the performance of the final model.

Method A: You should retrain the model on the full training set using the best parameters from the grid-search, but that doesn't mean you have to assume that the model doesn't over-fit. Note if you perform grid-search too thoroughly with too many hyper-parameters you can also over-fit the cross-validation error. The hyper-parameters are quite liklely to be over-fit to some extent, whether it causes a problem depends on the amount of data, the nature of the problem, how many grid points you use, the variance of the selection criterion, the number of hyper-parameters etc. Be skeptical and assume there will be problems.

Method B - indeed that makes poor use of the available data, and if the validation set is small, the uncertainty in estimating the threshold will be large.

Method C - same as B.

Method D - The optimal hyper-parameters for one threshold may not be the best hyper-parameters for another threshold. Easier just to view the threshold as a hyper-parameter and optimise them jointly.

Since my dataset is imbalanced I want to perform classification probability threshold tuning so I can find the optimal balance between sensitivity and specificity using G-mean over the returned values of the ROC-curves (explanation link).

A class imbalance doesn't mean that you need to perform probability threshold tuning. Whether you need that depends on the false-positive/false-negative misclassification costs (or alternatively the relative importance of sensitivity and specificity), which depend on the needs of the application and do not depend on the degree of imbalance. If you don't know either of those things, then you should probably be using a performance metric that doesn't depend on the threshold (such as the log-likelihood or Brier score or AUROC) as you don't have the relevant information to determine the threshold.

The most important thing is to think carefully about what is really important for your application and what prior knowledge you have about things like misclassification costs and build those into your evaluation procedure.