I have a question about precision and recall. Which classifier is better, one with high precision or high recall for medical purposes like finding patients with allergy? Can we use F-measure for finding better classifier ?
$\begingroup$
$\endgroup$
3
-
1$\begingroup$ Use neither. Everything here applies to precision, recall and F$\beta$: Why is accuracy not the best measure for assessing classification models? $\endgroup$– Stephan KolassaJan 15, 2019 at 7:09
-
$\begingroup$ It depends entirely on what you value more, precision or recall (and that depends on how you coded your factors). $\endgroup$– user2974951Jan 15, 2019 at 8:45
-
$\begingroup$ In addition to @StephanKolassa's comments which are spot on: not only do these proportions have inherent problems, but classifiers have a trade-off in where they put the class boundary. This means you can trade in sensitivity (recall) for higher specificity, and precision (Positive Predictive Value) against Negative Predictive Value. The bottomline is: what exactly is a good threshols (and thus, compromise between Sens vs. Spec and PPV vs. NPV) depends on your application. See e.g. stats.stackexchange.com/questions/312119/… $\endgroup$– cbeleites unhappy with SXJan 15, 2019 at 11:37
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
The choice for the classifier depends ultimately on the main goal you are trying to achieve!
One of the earliest steps to each project is to clearly understand what are the objectives and restrictions in which you will base your work upon.
Both Recall and Precision (and even the F-measure) suffer if your dataset is unbalanced.
Imagine you have a dumb classifier (let's call him derp) and derp(x) = 1, so you always predict the sample to be of a positive instance. If your true dataset has 150 positive and 50 negative instances, you will have recall = 1; precision = 0.75 and F ≃ 0.85 which is quite a good performance for such a dumb classifier.
For medical purposes, it depends on the severity of the issue in hands. If you are finding people with allergies I would advise minimizing the False Positive Rate, so that you do not alert too many people who may not have allergies. Nonetheless, for other more severe cases such as cancer prevention, I would advise minimizing the risk of not alerting a person which may be sick by minimizing the Miss Rate or False Negative Rate FNR = FN / P = 1 - Recall.
Also, you can check the ROC curve and it's Area Under Curve measure (see this) which can be a good metric to avoid dumb classifiers. For our derp AUC = 0.5 and this will always be the case for random or dumb classifiers, so you can check if your classifier is better than random if they have an AUC > 0.5.
Overall, the best advice I can give you is this:
Understand what you want and choose the best metric for it!
