Timeline for Has the journal Science endorsed the Garden of Forking Pathes Analyses?
Current License: CC BY-SA 3.0
8 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jul 12, 2016 at 3:14 | comment | added | Cliff AB | The results seem very consistent with doing two levels of cross-validation for model selection, but not using any holdout data for unbiased assessment of the final model. | |
| Jul 12, 2016 at 3:11 | comment | added | Cliff AB | Using the slides from your earlier link (icml.cc/2016/?page_id=97), on slides 72 and 73, even when using the "Thresholdout" method, the holdout accuracy is greater than the fresh data at every single simulation, although it does do better than "standard holdout" (which is really "standard abuse of the validation dataset", not an actual valid statistical procedure). FYI, the plot appears on the slides to be the same one in the Science paper (just in case you don't have access). | |
| Jul 12, 2016 at 2:06 | comment | added | horaceT | @CliffAB Can you elaborate? where? That'a an intriguing possibility.... | |
| Jul 12, 2016 at 0:43 | comment | added | Cliff AB | ...except that their own example is inconsistent with their claim of preventing over fitting, and is consistent with what I would expect the results to be from "I will only accept estimated effects $> \tau$". | |
| Jul 12, 2016 at 0:15 | history | edited | horaceT | CC BY-SA 3.0 |
added 190 characters in body
|
| Jul 12, 2016 at 0:13 | comment | added | horaceT | @CliffAB I have the same nagging feeling why this works better than just a simple threshold. But they have proofs! | |
| Jul 12, 2016 at 0:05 | comment | added | Cliff AB | But that's not an improvement over saying "I will only accept estimated effects $> \tau$"...which will not prevent over fitting (although it will slightly dampen it). Interestingly, in their own plots, you can see evidence of over fitting (systematically lower reported error on holdout data than on fresh data). | |
| Jul 11, 2016 at 23:31 | history | answered | horaceT | CC BY-SA 3.0 |