I trained a neural network to classify data. My data set consists of roughly 75% class 1 data and 25% class 2. After training, the network showed 84.4% accuracy. As the classes do not contain an equal amount of data, I also decided to look at the ROC curve. The last layer of my network uses a softmax, so that I can interpret the outputs as probabilities.
You find it in the following picture, blue line.
Additionally I plotted there the accuracy of the network vs. the threshold. So the x and y axes have to meanings here: for the blue curve, the x and y axes are the false positive rate and true positive rate, respectively. For the green curve, the x axis is the threshold and the y axis the accuracy of the network.
The blue points indicate the threshold/accuracy (FPR/TPR) if the threshold is chosen to be $T_1=0.5$. I also marked the point, which labels the highest accuarcy, namely 85.1% (red). The grey point is the one you get, when you minimize the distance between the ROC curve and the point (0,1). It corresponds to an accuracy of 82.2%.
Now to the questions:
- Is it in some way meaningful to change the threshold in order to maximize the accuracy or should one always stick to 0.5?
- If so, how should this be done? If one wants to tune the threshold, I assume that this has to be done with a held out data set and not with the test set. In this example, the criterion "minimize distance to (0,1)" delivers a result, which is worse than the standard 0.5 threshold. Just looking for a threshold which increases the accuracy seems at first like a good idead, but then it has nothing to do with the ROC curve. How does this curve come into play?
- What changes in the interpretation if I used a random forest? When I tried it, the curves looked very similar, i.e. a threshold slightly higher then 0.5 would yield higher accuracy.
- How does the area under the roc curve help me to increase the accuracy? Why is it even interesting to look at it? In the end one has one fixed threshold in the classifier, so why am I interested in how other thresholds perform?