1
$\begingroup$

I have a classification problem where getting true positives is much more important than true negatives.

To be clear, I know that roughly 10% of my population are actual positives, but I can assign some proportion (say 30%) of the population to be classified as positives without much cost, above all else I need to make sure that the actual positives are covered by this set.

Unfortunately the classification tools I am using in WEKA seem to be balancing precision and recall, so that it is a) not assigning as many positives as it is allowed to, and b) getting quite a bad recall value.

Is there a standard way of approaching this problem? My first guess would be to weight the cost function towards recall rather than F-score, but I don't see an easy way to do this in WEKA.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Optimizing recall in model selection will likely yield trivial classifiers that label everything as positive (perfect recall), so that won't help.

I suggest plotting ROC curves or Precision-Recall curves for your existing models to determine a decision threshold with the recall you desire. After that you can compare which models have the best specificity or precision (direct result of ROC and PR curves, respectively).

Improve this answer
$\endgroup$
7
  • $\begingroup$ Yeah I thought about something like this. Unfortunately I think I need to do some optimisation before the ROC stage - at the moment there are a lot of positives getting a very low score. $\endgroup$
    – sweezyjeezy
    Commented Feb 27, 2014 at 11:00
  • $\begingroup$ I agree that weighting 100% recall is a bad idea, but there are compromises, like the F_{beta} score - en.wikipedia.org/wiki/F1_score $\endgroup$
    – sweezyjeezy
    Commented Feb 27, 2014 at 11:01
  • 1
    $\begingroup$ Try weighting positives (or take multiple samples of them) to force a classifier to put more emphasis on positives. $\endgroup$
    – Marc Claesen
    Commented Feb 27, 2014 at 11:09
  • $\begingroup$ @sweezyjeezy However, the F1 score does not give higher weights towards Recall. It just assures that preference is given to procedures that prevent 0% recall and 100% precision to have a mean of 50%, compared to 50%-50%, which is often much preferred. $\endgroup$
    – PascalVKooten
    Commented Feb 27, 2014 at 11:09
  • $\begingroup$ Taking multiple samples is a good idea, I will try this $\endgroup$
    – sweezyjeezy
    Commented Feb 27, 2014 at 11:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.