Timeline for Risk score uncertainty quantification
Current License: CC BY-SA 4.0
10 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jun 9, 2022 at 12:00 | history | tweeted | twitter.com/StackStats/status/1534867927207378947 | ||
| Jun 6, 2022 at 15:58 | answer | added | Eike P. | timeline score: 4 | |
| May 28, 2022 at 18:26 | comment | added | Eike P. | @Dave Concerning your second question, I really need uncertainty quantification for individual risk predictions, not an overall model assessment. I edited the question to make this clearer. | |
| May 28, 2022 at 18:25 | comment | added | Eike P. | @Dave Well, what would you call an interval that contains the true risk with 95% probability then? I have been grappling with this terminology for a while now; right now it seems to me that it is really a "risk prediction interval" wrt the risk regression problem. This is different from a "prediction set" wrt a classifier built on a risk regression model , which would be either {0}, {0, 1} or {1}. But happy to learn if this is the wrong terminology? | |
| May 28, 2022 at 18:22 | comment | added | Eike P. | @whuber Thanks - you're right of course, the additive model did not make sense and is also not what I am actually assuming. I changed the formulation. However, I am really interested in general approaches, not in ways to do this for one specific model - hence my broad question. Nevertheless, I just asked a follow-up question on bootstrapping approaches specifically, and will probably draw out more sub-questions. I somehow expected the answer to this to be "Sure, you use the standard XYZ something interval, d'uh," hence my broad question. :-) | |
| May 28, 2022 at 18:18 | history | edited | Eike P. | CC BY-SA 4.0 |
added 447 characters in body
|
| May 24, 2022 at 16:15 | comment | added | Dave | A prediction interval gives an interval within which an observation is expected to be contained. A confidence interval gives an interval in which the mean of a distribution is expected to be contained. It doesn't make sense for a (binary) prediction interval to be something like $(0.3, 0.7)$, while that is a totally reasonable (even if wide) confidence interval. // What's wrong with having a measure of model performance? If the model tends to do well, then the predicted risk is believable. If the model tends to do poorly, then there is more uncertainty in the risk estimate. | |
| May 24, 2022 at 16:13 | comment | added | whuber♦ | "Canonical" might have some (heuristic, informal) meaning in mathematics, but is not relevant in statistics, where the model must be chosen based on circumstances and assumptions. Some things can be said about your approach, though, of which one of the most salient is that an additive error model for a probability like $r_i$ is unlikely to be suitable in most applications. Rather than asking such a generic question, then, please consider describing your actual problem. | |
| May 24, 2022 at 16:02 | history | edited | Eike P. | CC BY-SA 4.0 |
Added some further info on things I tried
|
| May 13, 2022 at 16:05 | history | asked | Eike P. | CC BY-SA 4.0 |