7
$\begingroup$

Consider a situation where we are running a classifier (the actual classification algorithm doesn't matter here), and the class labels are given based on a score. If score > 0, the data point is labeled A, if score < 0, the data point is labeled B.

All the training data contains data points with positive or negative scores. However, in my test data, there are a few points that return score = 0. How should I measure the precision and recall in this scenario where some points can't be classified into any class?

[additional information from a comment below] I faced this in a sentiment classification task. The normalized scores are in the range [-1,1], with 0 being the score for documents with no sentiment. It so happened that I had no neutral documents in my training data, but in the test data, some documents returned a score of 0.

Improve this question
$\endgroup$
6
  • $\begingroup$ Doesn't that simply mean that points with score = 0 are not counted in the set of retrieved items? en.wikipedia.org/wiki/… $\endgroup$
    – Mr E
    Commented Jul 20, 2012 at 15:51
  • $\begingroup$ In the classification context, I don't think we can say that items are retrieved. We need to think in terms of true-positive, false-positive, true-negative and false-negative. And once I get score = 0, I am not sure what it means in terms of these four categories. $\endgroup$
    – Chthonic Project
    Commented Jul 20, 2012 at 16:10
  • 2
    $\begingroup$ Either you have to decide for adding another label C which is unclassified or you have to use a default label, say A. But that is dependend on your domain knowledge. not that you classify people where you're unsure to have cancer ;) $\endgroup$
    – Thomas Jungblut
    Commented Jul 21, 2012 at 6:37
  • $\begingroup$ What does score "0" actually mean ? In default classification algorithms I know, there is no score zero. If the scores are generated by some sort of function, the probability of hitting one single real valued score is 0. So gaining a single score a significantly more often than once either means a dataset of "manageable complexity" or some explictly implemented special behavior. Which is ? $\endgroup$
    – steffen
    Commented Jul 24, 2012 at 7:11
  • $\begingroup$ @steffen : I faced this in a sentiment classification task. The normalized scores are in the range [-1,1], with 0 being the score for documents with no sentiment. It so happened that I had no neutral documents in my training data, but in the test data, some documents returned a score of 0. $\endgroup$
    – Chthonic Project
    Commented Jul 24, 2012 at 8:34

1 Answer 1

Reset to default
4
$\begingroup$

It's useful to keep in mind that precision/recall are inherently tied to a particular state or label of interest. In information retrieval that label might be "relevant" as opposed to "not relevant," whereas in cancer that label might be "malignant" as opposed to "benign."

As @Thomas Jungblut mentions, it would be valid to treat this not as a binary classification problem ("A" or "B") but instead as a multiclass classification problem ("A," "B," or "Unclassified"). There are other metrics besides precision/recall that can be of interest in multiclass classification. However, if you insist on precision/recall then you must pick your label of interest and then this sort of becomes de facto binary classification once again. You have various options for how to frame the problem ("A" vs "B or unclassified" is not the same as "A or unclassified" vs "B", etc.). However, effectively these are the same as simply picking a default label.

Since you seem to impart special meaning to the classification score of 0, it seems that perhaps it would be appropriate to also apply some domain knowledge or some knowledge of the specific classification algorithm being used. In the general case there's nothing really magical about a score of 0, but perhaps you really have a specific problem in mind where this is not the case.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.