# Accuracy of a random classifier

I was wondering how to compare accuracy of my classifier to a random one.

I'm going to elaborate further. Let's say we have a binary classification problem. We have $n^+$ positive examples and $n^-$ negative examples in the test set. I say that and record is positive with a probability of $p$.

I can estimate that, on average, I get:

\begin{align} TP &= pn^+ \\[2pt] TN &= (1-p)n^- \\[2pt] FN &= pn^- \\[2pt] FP &= (1-p)n^+ \end{align}

thus:

\begin{align} \mbox{acc} &= \frac{TP+TN}{TP+TN+FP+FN} \\[9pt] &= \frac{pn^+ + (1-p)n^-}{pn^+(1p)n^-pn^- + (1-p)n^+} \\[9pt] &= \frac{pn^+ + (1-p)n^-}{n^+ + n^-} \end{align}

For example if we have $n^+=n^-$ accuracy is always $1/2$ for any $p$.

This can be extended in multiclass classification: $$\mbox{acc} = \frac{\sum_{i=1}^c p_i n_i}{\sum n_i}$$ where $p_i$ is the probability to say "it is in the $i$th class", and $n_i$ is the count of records of class $i$. Also in this case, if $n_i = n/c \ \forall i$ then $$\mbox{acc} = 1/c$$

But, how can I compare the accuracy of my classifier without citing a test set? For example if I said: my classifier accuracy is 70% (estimated somehow, e.g. Cross-Validation), is it good or bad compared to a random classifier?

-

Thank you. You made me realize that you have always to take into account prior probabilities. Even if the accuracy is estimated by Cross-Validation, the classes do have prior probabilities. And, for example, you can exstimate them with $n_i$ that I wrote in my question. – Simone Oct 16 '11 at 10:01
You can get 90% of accuracy with $p=1$ in your problem. This gave me the insight to understand this: en.wikipedia.org/wiki/Gain_%28information_retrieval%29. You selected $p=1$ because you had seen the prior probability and you understood that it would have been better to be biased to the majority class. Thus, the best choise for a random classification is select $p=\frac{n^+}{n^+ +n^-}$ that gives the $r$ in wiki. Doing maieutics on my question I understood I wanted to know: if I flip a coin to classify, which is the $acc$? And it is always $1/2$, without care what the prior are. – Simone Oct 16 '11 at 10:19
Last comment. Classifying the positive class with probability $p$ means that your classifier lies along the main diagonal of the ROC curve (if you are able to compute it, because not all the classifier gives probabilities for output, as state in Kononenko and Bratko paper). Because $TPR = \frac{pn^+}{pn^+ + (1-p)n^+} = p$ and $FPR = \frac{pn^-}{pn^- + (1-p)n^-} = p$. With a diagonal ROC curve you can understand your performance is similar to a random one. Instead, changing $p$ changes $acc$, and it is somehow misleading. Thus, there should be better performance measures. – Simone Oct 16 '11 at 10:51