Problem statement A set of points is given. We want to classify those points in two distinct classes with labels $\{0,1\}$. We count the "hits" of the classifier based on the $1$ class. Assume that we have some classifiers in hand with the following properties

  • For some classifiers, the classification decision depends on some thresholds, which we may vary and plot a Precision/Recall or RoC curve
  • For others, the decision does not depend on any threshold but rather on a Property test. That is, we check if point $x_i$ has some property, and classify it accordingly. For illustration purposes one can imagine such a test as testing for example if $x_i$ is odd, and classifying all odd data points in the same class.(However this is not the problem I am working on!)

Additionally, we care more about misclassifying Label-$0$ points as Label-$1$ points (errors that would affect Precision), than misclassifying Label-$1$ points as Label-$0$ points(errors that affect Recall). To put it simply, we want the classified $1$ set to contain few outliers(Label-$0$ points) and we can sacrifice, to some extent, the size of the retrieved $1$-set.

Question Which measures could I use to evaluate the performance of each classifier and pick the classifier $c_i$ or the classifier/threshold pair $(c_i,T_i)$ that best solves the problem. Assume we have a dataset with known labels on which we evaluate the classifiers.

  • 1
    $\begingroup$ As a comment, I add this personal observation. For a given classifier, the AUC gives a measure of how well the classes are separated using a certain classifier. However, I would like to put more weight in Class' $0$ distribution entering Label $1$ area than the reverse scenario, and AUC puts equal weight in both cases. $\endgroup$
    – Prospects
    Feb 6 '14 at 17:11
  • $\begingroup$ well, then go one step back from the AUC and look at the acutal curve. Integrate only those parts that are OK according to your criteria. $\endgroup$ Feb 6 '14 at 20:09
  • $\begingroup$ @cbeleites if I understand you right, you suggest to integrate the error in the classes' distribution functions. However, the explicit form of the distributions is unknown, and only the true class labels in the test set are given. Thus, this method is inapplicable. Please clarify if I misunderstood the comment. $\endgroup$
    – Prospects
    Feb 7 '14 at 20:03
  • $\begingroup$ No, I just suggest that if you want to have some area under the (receiver-operating-)curve and have knowledge which regions of this curve are acceptable and which are not, then take into account only areas that are acceptable. In fact I think your concern is a huge step forward from the ususal rathe blind calculation of AUC - I think for most applications one type of error is more serious than the other. $\endgroup$ Feb 8 '14 at 12:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.