6
$\begingroup$

I have a testing data set where 1/3 of the observations are class-1 objects and the remainder class-0. Hence, the data set is skewed (skewed classifier), literature suggests that if the data set is skewed, use Precision Recall curves (AUC) and not the ROC-AUC.

For example, running the following code results in AUC_Pr_rand = 0.3267 and AUC_Pr_Ones = 0.3346. Where the first is a random classifier and the second a non-learning algorithm outputting ones only.

n=10000;
Y = rand(n,1) > 2/3;
Yi = rand(n,1);
[~,~,~,AUC_Pr_rand] = perfcurve(Y,Yi,true, 'xCrit', 'reca', 'yCrit', 'prec');
[~,~,~,AUC_Pr_Ones] = perfcurve(Y,1+rand(size(Yi))*0.00001,true, 'xCrit', 'reca',    'yCrit', 'prec');

A non-learning algorithm outputting just class-0, for the above testing set (1/3 are class-one) will give an accuracy of ~66%, but the above random classifier gives ~50% accuracy (I am not sure if this means anything, but I tested a logistic regression model and it gave an accuracy of ~65%, so I don't think accuracy is the metric to use for performance testing).

So now for any other given classifier (say, logistic regression or CART) tested on this skewed data-set, is the objective to beat this random classifier ie., obtain a PR-AUC above 0.3246 or 0.5 (because that is what an random classifier should output as PR-AUC for a balanced data set)?

$\endgroup$
  • $\begingroup$ Good question, I think the answer depends on the specifics of your application. – Correct if I'm wrong, but if you mean by "random classifier" that it doesn't use the feature information but just reproduces the label frequencies, its expected accuracy would be 1/3 * 1/3 + 2/3 * 2/3 = 5/9 = 55.55%, right? – Another performance measure for unbalanced data is balanced accuracy, i.e. the mean of the true positive and the true negative rate. Both for the always-0 classifier and the same-frequency classifier its value is 50%. $\endgroup$ – A. Donda Jan 20 '15 at 20:11
  • $\begingroup$ Reference for balanced accuracy: Brodersen et al., 2010 International Conference on Pattern Recognition, p. 3121, people.inf.ethz.ch/bkay/publications/Brodersen_2010b_ICPR.pdf $\endgroup$ – A. Donda Jan 20 '15 at 20:15
  • $\begingroup$ no I don't, otherwise I would've written an answer. ;-) But I think that the term "random classifier" is ambiguous and needs to be specified in order for you to get a correct answer. $\endgroup$ – A. Donda Jan 23 '15 at 15:03
4
+50
$\begingroup$

A random classifiers randomly selects a subset of the total data and labels it as positive. The size of said subset is associated with the recall of the random classifier. Since predictions are done entirely at random, the expected precision of such a labeling is equal to the fraction of positives in the total data set (at any recall). Hence, the PR curve of a random classifier is a horizontal line at precision=$\rho$ where $\rho$ is the fraction of positives in the total data set. The AUC is then immediately also equal to $\rho$.

In PR space the AUC of a random model is directly related to the class balance. An AUC of 0.5 can mean a tremendously good model for high class skew. Always compare PR-AUC for the given class skew, don't compare it to the balanced setting.

To answer your question: in general you do want to beat a random classifier, where random in PR space means having the curve I explained above.

In practice your objective depends entirely on what you want to do. Obviously, being worse than random is usually a very serious problem but it doesn't necessarily matter. For instance, if your application requires a model with high recall, you don't care if said model is worse than random at low recall.

$\endgroup$
  • 1
    $\begingroup$ What exactly does "randomly selects" mean? Won't the result be different depending on whether labels are selected e.g. with probability 0.5,0.5, or according to the observed proportion in the training data? $\endgroup$ – A. Donda Jan 23 '15 at 15:05
  • $\begingroup$ @A.Donda when computing a PR curve the proportion of instances that are assigned positive is varied between (0, 1]. For a given proportion, a random model assigns the right number of positive labels to data instances at random. $\endgroup$ – Marc Claesen Jan 23 '15 at 15:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.