Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 4 down vote favorite
1

I was wondering how to compare accuracy of my classifier to a random one.

I'm going to elaborate further. Let's say we have a binary classification problem. We have $n^+$ positive examples and $n^-$ negative examples in the test set. I say that and record is positive with a probability of $p$.

I can estimate that, on average, I get: $$TP = pn^+ \ TN=(1-p)n^- \ FN = pn^- \ FP = (1-p)n^+$$ thus $$\mbox{acc} = \frac{TP+TN}{TP+TN+FP+FN} = \frac{pn^+ + (1-p)n^-}{pn^+(1-p)n^-pn^- + (1-p)n^+}$$ that is $$ = \frac{pn^+ + (1-p)n^-}{n^+ + n^-}$$ For example if we have $n^+=n^-$ accuracy is always $1/2$ for any $p$.

This can be extended in multiclass classification: $$\mbox{acc} = \frac{\sum_{i=1}^c p_i n_i}{\sum n_i}$$ where $p_i$ is the probability to say "it is in the $i$th class", and $n_i$ is the count of records of class $i$. Also in this case, if $n_i = n/c \ \forall i$ then $$\mbox{acc} = 1/c$$

But, how can I compare the accuracy of my classifier without citing a test set? For example if I said: my classifier accuracy is 70% (estimated somehow, e.g. Cross-Validation), is it good or bad compared to a random classifier?

share|improve this question

1 Answer

up vote 1 down vote accepted

I am not sure I understand the last part of your question

But, how can I compare the accuracy of my classifier without citing a test set?

but I think I understand your concern. A given binary classifier's accuracy of 90% may be misleading if the natural frequency of one case vs the other is 90/100. If the classifier simply always chooses the most common case then it will, on average, be correct 90% of the time. A useful score to account for this issue is the Information score. A paper describing the score and its rationale can be found here. I learned about this score because it is part of the cross-validation suite in the excellent Orange data mining tools (you can use no-coding-needed visual programming or call libraries from Python).

share|improve this answer
Thank you. You made me realize that you have always to take into account prior probabilities. Even if the accuracy is estimated by Cross-Validation, the classes do have prior probabilities. And, for example, you can exstimate them with $n_i$ that I wrote in my question. – Simone Oct 16 '11 at 10:01
2  
You can get 90% of accuracy with $p=1$ in your problem. This gave me the insight to understand this: en.wikipedia.org/wiki/Gain_%28information_retrieval%29. You selected $p=1$ because you had seen the prior probability and you understood that it would have been better to be biased to the majority class. Thus, the best choise for a random classification is select $p=\frac{n^+}{n^+ +n^-}$ that gives the $r$ in wiki. Doing maieutics on my question I understood I wanted to know: if I flip a coin to classify, which is the $acc$? And it is always $1/2$, without care what the prior are. – Simone Oct 16 '11 at 10:19
Last comment. Classifying the positive class with probability $p$ means that your classifier lies along the main diagonal of the ROC curve (if you are able to compute it, because not all the classifier gives probabilities for output, as state in Kononenko and Bratko paper). Because $TPR = \frac{pn^+}{pn^+ + (1-p)n^+} = p$ and $FPR = \frac{pn^-}{pn^- + (1-p)n^-} = p$. With a diagonal ROC curve you can understand your performance is similar to a random one. Instead, changing $p$ changes $acc$, and it is somehow misleading. Thus, there should be better performance measures. – Simone Oct 16 '11 at 10:51
@Simone: +1 for finishing your thoughts and the useful link, but mostly for exposing me to a new word (maieutics). – Josh Hemann Oct 17 '11 at 4:53

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.