I have created four models of Logistic Regression (binary classification) and have developed the ROC curves of those 4 models. However, I am not able to understand how to scientifically determine the best model of the four and thereby finalize the best model of the four. I request someone to kindly help me out with determining the best reasoning to finalize the best model.
Let's go through some basics first and then expose the relevant information that the plot is conveying.
ROC curves are giving you a measure on the degree of separability of both classes by a model, that is, how well it can distinguish both classes. The curves that you see represent the respective
FPR (or percentages in your case) obtained by a model at different thresholds.
Understanding the role thresholds play here is important;
In order to compute the ROC curves, we need the probabilities output by the model, since the goodness of classification (measured in terms of
FPR) will be computed for different thresholds. Take for instance that the model is giving you a
0.8 probability. By using a
0.5 threshold, the sample would be classified as a
1, but instead it would be classified as a
0 with a
0.85 threshold. Checking the results through the corresponding metrics will result in a different point in the curve.
ROC curve of a classifier will be obtained by repeating the above, that is, computing the
FPR, on different thresholds until you get a line describing the general behaviour of the classifier.
But why do I need to know how well it classifies on
thresholds!=0.5? You may ask...
This will be telling you how good a classifier predicts in scenarios where you might not be all that interested in the best
AUC, but perhaps rather be more flexible with your
FPR for example (meaning going towards the right side of the graph, and getting a better
In that case you might want to compare the response of the different models at thresholds which have led to a higher
TPR (at the expense of a lower
FPR, and hence
AUC score). This will most likely not be occurring at a
0.5 threshold, and you'll have to see what the response is at different thresholds to see the setting that suits you the most.
In the general case, in which you want to minimise the mis-classification of both classes in a balanced way, the optimal threshold will the the one closer to the upper left corner, or in other words the one which results in a higher distance to the diagonal line (the diagonal line just represents the point at which a classifier does not know how to distinguish among both classes).
But as mentioned, what makes the
ROC curve so useful is that it tells you what is the goodness of classification of your model *depending on what metrics you want to prioritise
So to summarise, the information we can obtain from a
ROC curve as:
- What is the best threshold for each of the classifiers
- Which classifier behaves better overall
So essentially we can see it as a picture of how well a classifier behaves on different scenarios that might be of more or less interest according to each problem.