I have a question about precision and recall. Which classifier is better, one with high precision or high recall for medical purposes like finding patients with allergy? Can we use F-measure for finding better classifier ?
The choice for the classifier depends ultimately on the main goal you are trying to achieve!
One of the earliest steps to each project is to clearly understand what are the objectives and restrictions in which you will base your work upon.
Both Recall and Precision (and even the F-measure) suffer if your dataset is unbalanced.
Imagine you have a dumb classifier (let's call him derp) and
derp(x) = 1, so you always predict the sample to be of a positive instance. If your true dataset has 150 positive and 50 negative instances, you will have
recall = 1; precision = 0.75 and F ≃ 0.85 which is quite a good performance for such a dumb classifier.
For medical purposes, it depends on the severity of the issue in hands. If you are finding people with allergies I would advise minimizing the False Positive Rate, so that you do not alert too many people who may not have allergies. Nonetheless, for other more severe cases such as cancer prevention, I would advise minimizing the risk of not alerting a person which may be sick by minimizing the Miss Rate or False Negative Rate
FNR = FN / P = 1 - Recall.
Also, you can check the ROC curve and it's Area Under Curve measure (see this) which can be a good metric to avoid dumb classifiers. For our derp
AUC = 0.5 and this will always be the case for random or dumb classifiers, so you can check if your classifier is better than random if they have an
AUC > 0.5.
Overall, the best advice I can give you is this:
Understand what you want and choose the best metric for it!