Timeline for Training on predicted labels for cross validation
Current License: CC BY-SA 4.0
7 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jan 29, 2023 at 13:09 | history | bounty ended | CommunityBot | ||
| Jan 27, 2023 at 16:58 | comment | added | Blaze | Marking as the answer for the links provided. Much appreciated. | |
| Jan 27, 2023 at 16:43 | vote | accept | Blaze | ||
| Jan 27, 2023 at 16:41 | comment | added | Blaze | "your model errors are highly statistically dependent (almost perfectly correlated?) b/c the second model is fit on predictions from the first." This is one of the reasons I look for references. These are all shallow, off the cuff responses without any careful investgiation or study of what is being proposed. As I mentioned, when you change sample sizes or distribution, the correlation breaks down so your reasoning is easily proven false. | |
| Jan 27, 2023 at 16:31 | comment | added | Blaze | "So for that example, and in the context of out-of-distribution error estimation, it's wrong to be concerned about ignoring unlabeled data." You can roughly simulate out of distribution data by reducing your sample sizes. Try running the above code with 0.001 sample sizes for train/holdout, for example. CV is reduced to noise, but SSL metrics works great. | |
| Jan 27, 2023 at 15:37 | comment | added | Blaze | Sometimes all you have is predictions, so CV isn't necessarily the baseline to beat. For example, imagine you're inviting people to submit models and want to validate them before they share code. | |
| Jan 27, 2023 at 9:13 | history | answered | chicxulub | CC BY-SA 4.0 |