3
$\begingroup$

In binary classification, are there criteria or guidelines available to judge if classification performance of the testset (unseen data) is poor, medium or high?

I realise that this may depend on the research context, so perhaps there is not a straight answer to my question. For example, in medicine high classification performance may be critical for patient care while in stock markets lower classification performance may be acceptable if that means financial gain. However, if looking at measures of sensitivity, specificity, accuracy (and positive and negative predictive value as well), one wants these values to be as high as possible for the testset. Or otherwise, should the area under the curve of the ROC surpass a certain threshold to be considered sufficient (for ex. 0.8).

My question is: Are there general guidelines available which levels are acceptable or does this depend entirely on the research topic one is working on?

$\endgroup$

3 Answers 3

1
$\begingroup$

My bet would be on "common sense" or more precisely "business value". What's the difference between right and wrong classifications that would harm your business? You mentioned the medical domain. Is it acceptable for your company to put in danger 10% of your patients? Or if you want to predict changes in price of shares is a difference of 10% in growth/reduction that important? So, in my opinion, the performance of your classifier is either good or bad depending on the domain (what's acceptable in the real world).

$\endgroup$
1
  • $\begingroup$ (+1) Good point for not only reporting some general statistics, business value of decision and/or risk must be included also. $\endgroup$
    – Analyst
    May 12, 2015 at 5:00
1
$\begingroup$

I use mostly AUC statistic and shape of ROC - curve in test partition to see if my model can truly classify cases. I like algorithms which have probabilistic output, since after this I can personally change threshold T, which I use when making decision.

Suppose that in telemarketing campaign it is quite costly to make an call to the subject and the response rate Q is very low generally. Then it might be worthwhile to adjust T in such way that T some big multiple of Q.

Short answer: it is industry or subject specific matter what is good or acceptable classification accuracy.

$\endgroup$
1
  • $\begingroup$ Thank you! Are there any other statistics you would suggest apart from area under curve? $\endgroup$ May 12, 2015 at 8:21
0
$\begingroup$
  1. Confusion matrix: This helps to identify where exactly the classification is erring. You can also manually assign weights to the results

  2. Kappa Value: Anything above 0.8 is considered good. 0.9 is excellent for the model.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.