12
$\begingroup$

I have 2-classes labelled data on which I'm performing classification using multiple classifiers. And the datasets are well balanced. When assessing the classifiers' performance, I need to take into consideration how accurate the classifier is in determining not only the true positives, but the true negatives also. Therefore, if I use accuracy, and if the classifier is biased toward positives and classifies everything as positive, I will get around 50% accuracy, even though it failed at classifying any true negatives. This property is extended to precision and recall as they focus on one class only, and in turn to F1-score. (This is what I understand even from this paper for example "Beyond Accuracy, F-score and ROC: a Family of Discriminant Measures for Performance Evaluation").

Therefore, I can use sensitivity and specificity (TPR and TNR) to see how the classifier performed for each class, where I aim to maximize these values.

My question is that I am looking for a measure that combines both these values into one meaningful measure. I looked into the measures provided in that paper, but I found it to be non-trivial. And based on my understanding I was wondering why can't we apply something like the F-score, but instead of using precision and recall I would use sensitivity and specificity? So the formula would be $$ \text{my Performance Measure} = \frac{2 * \text{sensitivity} * \text{specificity}}{\text{sensitivity} + \text{specificity}} $$ and my aim would be to maximize this measure. I find it to be very representative. Is there a similar formula already? And would this make sense or is it even mathematically sound?

Improve this question
$\endgroup$

3 Answers 3

Reset to default
2
$\begingroup$

I would say that there might not be any particular or only one measure which you should take into account.

Last time when I did probabilistic classification I had a R package ROCR and explicit cost values for the False Positives and False Negatives.

I considered all cutoff-points from 0 to 1 and used many measures such as expected cost when selecting this cutoff - point. Of course I had already AUC measure for the general measure of classifying accuracy. But for me this was not the only possibility.

Values for the FP and FN cases must come outside your particular model, maybe these are provided by some subject matter expert?

For example in customer churn analysis it might be more expensive to incorrectly infer that customer is not churning but also that it will be expensive to give a general reduction in prices for services without accurary to target these to correct groups.

-Analyst

Improve this answer
$\endgroup$
1
  • $\begingroup$ Actually for my case it is sort of similar. Because FP and FN cases are gonna be costly in my model. I eventually ended up doing something similar to what you suggested "using multiple measures". I calculated the F-score for each class label, and to assess the models I use both these values along with some cost function that uses precision (for both classes) to calculate profit and subtracts from it the loss incurred from FP and FN cases. $\endgroup$
    – Kalaji
    Sep 1, 2013 at 0:21
4
$\begingroup$

Classification accuracy, sensitivity, specificity, and any simple combination of them are all improper scoring rules. That is, they are optimized by a bogus model. Using them will make you choose the wrong features, give the wrong weights, and make suboptimal decisions. One of many ways decisions are suboptimal is the false confidence you get when predicted probabilities are near the threshold implied by the use of these measures. In short, everything that can go wrong does go wrong with these measures. Using them to compare even two well-fitted models will mislead you.

Improve this answer
$\endgroup$
6
  • 3
    $\begingroup$ I agree that any generated model is a "bogus model" as you mentioned. But still I need a measure to assess its quality, to choose a model eventually. Assuming that my features have a already been selected (trying multiple datasets with different sets of features), and I am using 5-fold cross validation in order to determine whether my classifiers are overfitting the data, these simple "scoring rules" are the most widely used in literature. What other measures would you suggest then? Most of the measures rely on combinations of these values including LR+/-, ROC, and AUC. $\endgroup$
    – Kalaji
    Aug 10, 2013 at 23:13
  • $\begingroup$ First of all are you careful to repeat all exploratory/modeling steps from scratch for each of the 5 model fits used in 5-fold cv? The gold standard quality measure is the log likelihood and quantities derived from it such as $R^2$ and deviance. For binary $Y$ this leads to a logarithmic probability scoring rule. For that case you can also use another proper score, the Brier score (mean squared error in predicted probabilities). $\endgroup$
    – Frank Harrell
    Aug 11, 2013 at 3:59
  • $\begingroup$ Based on my reading this applies in case my models generate probabilities rather than discrete values (i.e. a probability that an instance belongs to class 0 or 1 instead of outputting 0 or 1). And in turn, this had to do with the classifiers implementation, e.g. it applies to a Naive Bayes classifier but not to a 1-NN classifier. Notice that I am not implementing the classifiers, I'm using some classifiers in Weka to generate my models. Maybe I'm a bit confused here. Thanks. $\endgroup$
    – Kalaji
    Aug 11, 2013 at 15:50
  • 1
    $\begingroup$ If the method you are using does not yield probabilities I suggest finding another method. $\endgroup$
    – Frank Harrell
    Aug 11, 2013 at 17:31
  • $\begingroup$ If there are well-understood disparities between the actual cost of precision and sensitivity (not applicable to the original post), why would you avoid using those? Would a biased cross-entropy-error be preferable (e.g., the penalty of the (1-c)*log(1-p) term is doubled)? $\endgroup$
    – Max Candocia
    Jun 28, 2017 at 21:36
0
$\begingroup$

EDIT: Whoops didn't realise how old this was, hope this is useful for anyone who stumbles upon it

You've got a bunch of options which make sense depending on what fits your exact context:

  • Balanced accuracy is just the mean of sensitivity and specificity, but still usually preferable to accuracy
  • F1 is the harmonic mean of sensitivity and PPV, and as you said is what you want when you care about the positive class, but not the negative class so much
  • MCC is more balanced than F1, and may be what you're after: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7863449/

In terms of general advice - perhaps interrogate what your context demands a little bit more. What is more harmful - FN or FP? You can tailor which metrics suit your need a little more, and a holistic approach to what they all represent (in words even) would be more sensible than relying on a single value.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.