1
$\begingroup$

I have some imbalanced data for binary classification, which I have preprocessed in 2 different ways. That led to having a different number of observations and pos/neg ratio. Then I trained the same model (same parameters, except class weights), let's say Logistic Regression, and have achieved the next results:

AUPRC Baseline delta
0.34 0.22 0.12
0.43 0.27 0.16

delta == AUPRC - Baseline

AUPRC is an area under Precision-Recall curve.

It’s a bit trickier to interpret AUPRC than it is to interpret AUROC (the area under the receiver operating characteristic). That’s because the baseline for AUROC is always going to be 0.5 — a random classifier, or a coin toss, will get you an AUROC of 0.5. But with AUPRC, the baseline is equal to the fraction of positives, where the fraction of positives is calculated as (# positive examples / total # examples).

  1. Is it legitimate to say, looking at deltas, that the second case is better than the first one? -- 0.12 vs 0.16
  2. Can I say that the second case is 33% more accurate than the first one? -- 0.16 / 0.12 - 1 ~= 0.33

The main idea is to show that more accurate data preprocessing leads to better target metric results (more "accurate" or "precise").

Improve this question
$\endgroup$
2
  • $\begingroup$ Not sure what ROC AUPRC is? (A combo of precision-recall and ROC curves?) In general for 2 I think the answer is no -- neither curve directly translates to 'accuracy' as generally defined # correct/total cases. If one dominates the other (e.g. they never cross) a variant of saying model 2 is better than model 1 though would be easier to make I think. $\endgroup$
    – Andy W
    Commented Jun 24, 2021 at 12:58
  • $\begingroup$ @AndyW yep, Precision-Recall curve: glassboxmedicine.com/2019/03/02/measuring-performance-auprc $\endgroup$
    – banderlog013
    Commented Jun 24, 2021 at 22:58

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.