Timeline for Why do we need to add confidence intervals to model predictions in cases we don't know the true data distribution?
Current License: CC BY-SA 4.0
11 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Nov 15, 2022 at 10:57 | comment | added | Christian Hennig | If your data have extreme outliers (as any not very small sample from the Cauchy will have), you shouldn't just compute means in the first place, but rather go for something more robust. | |
| Nov 15, 2022 at 10:55 | comment | added | Christian Hennig | The Cauchy distribution is not a "typical" non-Gaussian distribution, but is outright evil. Things that lead to disaster with Cauchy can still work reasonably well with all kinds of other non-Gaussian distributions for which moments exist. In reality almost all measurements are bounded in some way so that moments will in fact exist (although there can of course be extreme outliers, which need to be dealt with). | |
| Nov 15, 2022 at 10:09 | answer | added | Stephan Kolassa | timeline score: 1 | |
| Nov 14, 2022 at 21:48 | comment | added | partizanos | @StephanKolassa I added an example | |
| Nov 14, 2022 at 21:47 | history | edited | partizanos | CC BY-SA 4.0 |
added a subquestion to make the question more focused for an actual case where sampling is common practice
|
| Nov 14, 2022 at 21:36 | comment | added | partizanos | Yes and I see it in quite some papers I will put an example in the question to demonstrate the case. | |
| Nov 14, 2022 at 19:42 | comment | added | Stephan Kolassa | That is yet a third possible meaning: a range of point predictions for different randomizations. This is neither a confidence interval nor a prediction interval. Is it this you are asking about? | |
| Nov 14, 2022 at 19:08 | comment | added | partizanos | My question is more about ML model predictions. ie people report scores +/- for different instances of the model configured with the same hyperparameters but different random seeds to initialize weights and random states in various parts. | |
| Nov 14, 2022 at 11:04 | answer | added | jcken | timeline score: 2 | |
| Nov 14, 2022 at 10:53 | comment | added | Stephan Kolassa | Are you asking about confidence intervals or prediction intervals? There is a difference. I don't think many people give PIs for Cauchys, so it seems like you are looking at CIs... but then again, "mean and variance" does sound more like PIs. | |
| Nov 14, 2022 at 10:48 | history | asked | partizanos | CC BY-SA 4.0 |