0
$\begingroup$

I have created four models of Logistic Regression (binary classification) and have developed the ROC curves of those 4 models. However, I am not able to understand how to scientifically determine the best model of the four and thereby finalize the best model of the four. I request someone to kindly help me out with determining the best reasoning to finalize the best model.enter image description here

$\endgroup$
1
$\begingroup$

Let's go through some basics first and then expose the relevant information that the plot is conveying.


The ROC curves are giving you a measure on the degree of separability of both classes by a model, that is, how well it can distinguish both classes. The curves that you see represent the respective TPR and FPR (or percentages in your case) obtained by a model at different thresholds.

Understanding the role thresholds play here is important; In order to compute the ROC curves, we need the probabilities output by the model, since the goodness of classification (measured in terms of TPRvsFPR) will be computed for different thresholds. Take for instance that the model is giving you a 0.8 probability. By using a 0.5 threshold, the sample would be classified as a 1, but instead it would be classified as a 0 with a 0.85 threshold. Checking the results through the corresponding metrics will result in a different point in the curve.

The entire ROC curve of a classifier will be obtained by repeating the above, that is, computing the TPR and FPR, on different thresholds until you get a line describing the general behaviour of the classifier.

But why do I need to know how well it classifies on thresholds!=0.5? You may ask...

This will be telling you how good a classifier predicts in scenarios where you might not be all that interested in the best AUC, but perhaps rather be more flexible with your FPR for example (meaning going towards the right side of the graph, and getting a better TPR).

In that case you might want to compare the response of the different models at thresholds which have led to a higher TPR (at the expense of a lower FPR, and hence AUC score). This will most likely not be occurring at a 0.5 threshold, and you'll have to see what the response is at different thresholds to see the setting that suits you the most.

In the general case, in which you want to minimise the mis-classification of both classes in a balanced way, the optimal threshold will the the one closer to the upper left corner, or in other words the one which results in a higher distance to the diagonal line (the diagonal line just represents the point at which a classifier does not know how to distinguish among both classes).

But as mentioned, what makes the ROC curve so useful is that it tells you what is the goodness of classification of your model *depending on what metrics you want to prioritise

So to summarise, the information we can obtain from a ROC curve as:

  • What is the best threshold for each of the classifiers
  • Which classifier behaves better overall

So essentially we can see it as a picture of how well a classifier behaves on different scenarios that might be of more or less interest according to each problem.

$\endgroup$
1
  • 1
    $\begingroup$ Thank you Yatu for your response. Helped me a lot.😊 $\endgroup$ – Kartik Rayaprolu Apr 22 '20 at 4:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.