I'm busy evaluating a classifier and noticed something odd. The error rate seems to be always low, however the False Alarm Rate are in some instances > 90%. Where the Error rate are 30% the FAR reaches 44%.

How does this make sense?

Sampled confusion matrix:

TP 2 400 FP 244 TN 9 FN 10

  • $\begingroup$ TP 2 400 FP 244 TN 9 FN 10 $\endgroup$ – user2459813 Jan 18 '17 at 10:14
  • $\begingroup$ Please add this information properly formatted into the question. $\endgroup$ – Nikolas Rieble Jan 18 '17 at 11:18

How are the true and false samples distributed in your test and train data?

In case of high imbalance, a classifier which always predicts "true" could yield an of 99% (error rate of 1%) and a false alarm rate of 100%.

I therefore assume, that the data is imbalanced. Instead of using the accuracy (error rate), you could evaluate using the area under the curve (AUC). For more information, see here.

  • $\begingroup$ Thank you that makes a-lot of sense! How would one then work around this, since the datasets are unfortunately unbalanced, since it's with decision trees the ROC analysis would not be usable since there's no threshold to adjust?I've added some of the confusion matrix data. $\endgroup$ – user2459813 Jan 18 '17 at 10:12
  • $\begingroup$ You could compute a loss as the sum of relative accuracy per class. Then train using this as a loss function. $\endgroup$ – Nikolas Rieble Jan 18 '17 at 11:18
  • $\begingroup$ (+1) But with regard to your comment, shouldn't a loss function be got from the real losses incurred by each type of mis-classification? Who's to say that the classifier that always predicts "true" isn't doing a good job, albeit an easy one? $\endgroup$ – Scortchi Jan 18 '17 at 11:41
  • 1
    $\begingroup$ It depends on the user to define the loss-function, as the loss function defines the target behaviour. If the user wants high accuracy, then a "normal" loss function is fine, yet if the user specifically wants low errorrate and low False-Alarm-Rate, then the loss-function will have to be adjusted. Whether or not a job is done well, only the person defining the task/target is able to decide. This is where the importance of choosing/adjusting the loss function comes in. $\endgroup$ – Nikolas Rieble Jan 18 '17 at 11:47
  • 1
    $\begingroup$ I think we agree then. $\endgroup$ – Scortchi Jan 18 '17 at 11:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.