Timeline for Why does the mean AURPC go down the more examples one uses?
Current License: CC BY-SA 4.0
8 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Aug 17, 2023 at 6:39 | vote | accept | Tobias Hermann | ||
| Aug 17, 2023 at 6:39 | comment | added | Tobias Hermann | Ah, I think I get it now. Thanks a lot! | |
| Aug 15, 2023 at 23:27 | comment | added | TheD | You can also see this if you take your original betas and use the b_neg for the positive samples essentially making a classifier that has inverted predictions. Your PR AUC will now be closer to the lowest PR aucs given the number of samples but still be decreasing because adding more samples allows for smaller average precisions. | |
| Aug 15, 2023 at 23:27 | comment | added | TheD | If your question is why it always goes from a higher PR AUC to a lower one: If you have two samples, your PR AUC is 1 if you get it right and 0.5(not 0) if you get it wrong so depending on your distribution details you are somewhere between the two (averaging many trials). If you have four samples (with two 1 and two 0) the lowest PR AUC for any random sample is 0.416 etc. So in your sampling you always have perfect AUC (1.0) but you never get a score like 0. | |
| Aug 15, 2023 at 23:27 | comment | added | TheD | I am not sure I understand what you mean, in your new example the probability that B_neg<B_pos is 0.48065 so the cases in ranking we have for N=1 is b_neg b_pos (PR AUC is 1, that happens (1-0.48065) of the time) b_pos b_neg (PR AUC is 0.5 that happens 0.48065 of the time) So the PR AUC is 1*(1-0.48065)+0.5*0.48065 ~ 0.759 which is similar to your N=1 case in the code | |
| Aug 15, 2023 at 12:40 | comment | added | Tobias Hermann | Thanks for the answer. I think it makes sense when looking at single evaluations, especially with a sample size of 1. But we evaluate 10000 times (and the mean of the AUPRC is calculated) I wonder why the probable case (high AURPC) does not even out with the rare case (low AURPC) on average, approaching the expected value (without overestimation bias). I just tried with two distributions with a very strong overlap (low expected AURPC), expecting that this might invert the effect, but it does not. | |
| S Aug 15, 2023 at 1:14 | review | First answers | |||
| Aug 15, 2023 at 1:22 | |||||
| S Aug 15, 2023 at 1:14 | history | answered | TheD | CC BY-SA 4.0 |