2
$\begingroup$

I've been trying to understand how to interpret what random performance would be for a model I have on the AUPRC score. By 'random performance' I mean the worst possible performance. Purely unintelligent, meaning the model has learned nothing.

With AUPRC for example, random performance leads to a score of 0.5. If you get AUC <= 0.5, then your model serves no use.

With AUPRC, I have learned that this base performance you want to beat is equal to the number of positive samples divided by the total number of samples (for example in this post). However I still don't understand why this is the worst case performance, and I can't find anywhere that explains this.

A concrete example: Let's say I have 20 samples of Class 0 for every sample of Class 1. Then according to the above post, the worst possible performance I could get on AUPRC (where class 1 is the positive class) is 1/21 = 0.047...

My Question:

Why is 1/21 the worst performance for AUPRC in this case?

$\endgroup$

1 Answer 1

0
$\begingroup$

The rationale is as follows:

If you know nothing about the data other than the fact that there are $20$ instances of category $0$ for every one instance of category $1$, the most sensible prediction about the probability of a new observation belonging to category $1$ is $1/21$.

This kind of comparison to a naïve model is routine in other forms of modeling. For instance, the usual $R^2$ in OLS linear regression can be seen is a comparison of your model performance to the performance of a model that predicts the overall mean every time. The approach here is the same: use the prior probability as your best guess of the posterior probability, since you have no features to help you make a better prediction of the posterior probability.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.