Change threshold of classifier based on ROC

Question

I trained a neural network to classify data. My data set consists of roughly 75% class 1 data and 25% class 2. After training, the network showed 84.4% accuracy. As the classes do not contain an equal amount of data, I also decided to look at the ROC curve. The last layer of my network uses a softmax, so that I can interpret the outputs as probabilities.

You find it in the following picture, blue line.

Additionally I plotted there the accuracy of the network vs. the threshold. So the x and y axes have to meanings here: for the blue curve, the x and y axes are the false positive rate and true positive rate, respectively. For the green curve, the x axis is the threshold and the y axis the accuracy of the network.

The blue points indicate the threshold/accuracy (FPR/TPR) if the threshold is chosen to be $T_1=0.5$. I also marked the point, which labels the highest accuarcy, namely 85.1% (red). The grey point is the one you get, when you minimize the distance between the ROC curve and the point (0,1). It corresponds to an accuracy of 82.2%.

Now to the questions:

1. Is it in some way meaningful to change the threshold in order to maximize the accuracy or should one always stick to 0.5?
2. If so, how should this be done? If one wants to tune the threshold, I assume that this has to be done with a held out data set and not with the test set. In this example, the criterion "minimize distance to (0,1)" delivers a result, which is worse than the standard 0.5 threshold. Just looking for a threshold which increases the accuracy seems at first like a good idead, but then it has nothing to do with the ROC curve. How does this curve come into play?
3. What changes in the interpretation if I used a random forest? When I tried it, the curves looked very similar, i.e. a threshold slightly higher then 0.5 would yield higher accuracy.
4. How does the area under the roc curve help me to increase the accuracy? Why is it even interesting to look at it? In the end one has one fixed threshold in the classifier, so why am I interested in how other thresholds perform?

Answer 1

The Accuracy (or the area under the ROC curve) depends on the sample used to construct the ROC curve (see, e.g., How to interpret 95% confidence interval for Area Under Curve of ROC?). So if you use this ROC to optimise the threshold there is a risk of overfitting to the sample.

You can indeed "calibrate" a threshold using a hold-out data set. You might also use k-fold cross validation. The link with the ROC-curve is the fact that that there exists a link between the area under the ROC-curve and the accuracy ratio ($AR=2AUC-1$).

The Area under the ROC (AUROC or AUC) is linked to the accuracy ratio. The area under the curve can be interpreted as follows: of all possible pairs with one subject of class 1 and one subject of class 2, there is a fraction equal to AUC for which the subject in class 1 will have a better score.

This interpretation is the same when you use a random forest.

You can find more on all this in, e.g., this paper