# Is the objective to beat a random classifier when the data set is skewed using PR curves?

I have a testing data set where 1/3 of the observations are class-1 objects and the remainder class-0. Hence, the data set is skewed (skewed classifier), literature suggests that if the data set is skewed, use Precision Recall curves (AUC) and not the ROC-AUC.

For example, running the following code results in AUC_Pr_rand = 0.3267 and AUC_Pr_Ones = 0.3346. Where the first is a random classifier and the second a non-learning algorithm outputting ones only.

n=10000;
Y = rand(n,1) > 2/3;
Yi = rand(n,1);
[~,~,~,AUC_Pr_rand] = perfcurve(Y,Yi,true, 'xCrit', 'reca', 'yCrit', 'prec');
[~,~,~,AUC_Pr_Ones] = perfcurve(Y,1+rand(size(Yi))*0.00001,true, 'xCrit', 'reca',    'yCrit', 'prec');


A non-learning algorithm outputting just class-0, for the above testing set (1/3 are class-one) will give an accuracy of ~66%, but the above random classifier gives ~50% accuracy (I am not sure if this means anything, but I tested a logistic regression model and it gave an accuracy of ~65%, so I don't think accuracy is the metric to use for performance testing).

So now for any other given classifier (say, logistic regression or CART) tested on this skewed data-set, is the objective to beat this random classifier ie., obtain a PR-AUC above 0.3246 or 0.5 (because that is what an random classifier should output as PR-AUC for a balanced data set)?

• Good question, I think the answer depends on the specifics of your application. – Correct if I'm wrong, but if you mean by "random classifier" that it doesn't use the feature information but just reproduces the label frequencies, its expected accuracy would be 1/3 * 1/3 + 2/3 * 2/3 = 5/9 = 55.55%, right? – Another performance measure for unbalanced data is balanced accuracy, i.e. the mean of the true positive and the true negative rate. Both for the always-0 classifier and the same-frequency classifier its value is 50%. – A. Donda Jan 20 '15 at 20:11
• Reference for balanced accuracy: Brodersen et al., 2010 International Conference on Pattern Recognition, p. 3121, people.inf.ethz.ch/bkay/publications/Brodersen_2010b_ICPR.pdf – A. Donda Jan 20 '15 at 20:15
• no I don't, otherwise I would've written an answer. ;-) But I think that the term "random classifier" is ambiguous and needs to be specified in order for you to get a correct answer. – A. Donda Jan 23 '15 at 15:03

A random classifiers randomly selects a subset of the total data and labels it as positive. The size of said subset is associated with the recall of the random classifier. Since predictions are done entirely at random, the expected precision of such a labeling is equal to the fraction of positives in the total data set (at any recall). Hence, the PR curve of a random classifier is a horizontal line at precision=$\rho$ where $\rho$ is the fraction of positives in the total data set. The AUC is then immediately also equal to $\rho$.