1
$\begingroup$

I am computing the precision-recall curve for my ML model with 2 classes. I want to have a p-value that compare the observed area under the precision recall curve (AUPR_obs) and the area of such a curve with random labels (AUPR_rand). I know that the random area under the precision recall would be close to the proportion of the positive class. What method can I use to obtain the p-value comparing AUPR_obs and AUPR_rand? Would DeLong test be a solution?

$\endgroup$
1
  • 1
    $\begingroup$ I suggest avoiding presenting the confidence in a classifier's performance by giving a $p$-value for a performance metric; a $p$-value is "the probability, given a null hypothesis for the probability distribution of the data, that the outcome would be as extreme as, or more extreme than, the observed outcome". A random classifier is a nonsensical null hypothesis. It is an unreasonable benchmark. If our classifier is say a RF with some feature engineering we should show we are consistently better than a Logistic regression with no feature engineering. We don't "classify by chance". $\endgroup$
    – usεr11852
    Apr 17, 2021 at 21:45

2 Answers 2

1
$\begingroup$

One of the simplest solutions to problems where one needs confidence intervals or hypothesis tests for non-standard metrics (for which there may be no ready to go solution out there), is to use bootstrapping - which should be relatively easy in the case you describe (I assume there is just a single ML model, a single "ground truth" and a reasonable sized test set that you are working with). There are some situations where bootstrapping gets really difficult (e.g. multiple readers look at X-ray images of multiple patients, but each patient gets look at by several readers, in that case simple bootstrapping does not work), but this scenario seems straightforward.

You randomly draw observations with replacement calculate the difference between the two areas under the precision recall curves and then calculate standard error and test statistic. For the one using a random predictors (i.e. predicting uniform random numbers between 0 and 1), you probably should not actually simulate the predictons (that just adds an extra source of simulation uncertainty), but rather calculate the performance directly (AUPRC for a predictor that predicts i.i.d. U(0,1) values is just the proportion of the target class in the bootstrap sample, I believe).

Unless the dataset is very large, calculating the difference in AUPRC between the ML Model and a random classifier for each bootstrap samples should be incredibly fast, so you can likely afford to do a pretty large number of boostrap samples (at least so large that the randomness of boostrapping no longer plays any meaningful role).

$\endgroup$
0
$\begingroup$

I recently faced a similar issue and implemented a bootstrap-based method to test the hypothesis that the real difference in PR AUCs is different than zero, see the R package usefun.

For your case, I agree that probably you shouldn't do that if one of the two PR curves is the random classifier, it's a nonsensical benchmark. Also note that DeLong test is only used to compare ROC curves.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.