I have a testing data set where 1/3 of the observations are class-1 objects and the remainder class-0. Hence, the data set is skewed (skewed classifier), literature suggests that if the data set is skewed, use Precision Recall curves (AUC) and not the ROC-AUC.
For example, running the following code results in AUC_Pr_rand = 0.3267 and AUC_Pr_Ones = 0.3346. Where the first is a random classifier and the second a non-learning algorithm outputting ones only.
n=10000; Y = rand(n,1) > 2/3; Yi = rand(n,1); [~,~,~,AUC_Pr_rand] = perfcurve(Y,Yi,true, 'xCrit', 'reca', 'yCrit', 'prec'); [~,~,~,AUC_Pr_Ones] = perfcurve(Y,1+rand(size(Yi))*0.00001,true, 'xCrit', 'reca', 'yCrit', 'prec');
A non-learning algorithm outputting just class-0, for the above testing set (1/3 are class-one) will give an accuracy of ~66%, but the above random classifier gives ~50% accuracy (I am not sure if this means anything, but I tested a logistic regression model and it gave an accuracy of ~65%, so I don't think accuracy is the metric to use for performance testing).
So now for any other given classifier (say, logistic regression or CART) tested on this skewed data-set, is the objective to beat this random classifier ie., obtain a PR-AUC above 0.3246 or 0.5 (because that is what an random classifier should output as PR-AUC for a balanced data set)?