Timeline for Formal treatment of overfitting on the test set
Current License: CC BY-SA 4.0
7 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Oct 15, 2021 at 20:42 | history | edited | Sextus Empiricus | CC BY-SA 4.0 |
added 50 characters in body
|
| Oct 15, 2021 at 6:45 | history | edited | Sextus Empiricus | CC BY-SA 4.0 |
added 740 characters in body
|
| Oct 15, 2021 at 6:40 | history | edited | Sextus Empiricus | CC BY-SA 4.0 |
added 740 characters in body
|
| Oct 14, 2021 at 20:24 | comment | added | Sextus Empiricus | So, if the difference between testing and cross validation is clear, then what is unclear? | |
| Oct 14, 2021 at 20:23 | comment | added | Sextus Empiricus | @Sanyo maybe I am not seeing the problem so clearly. If you use the test set as some sort of second layer of cross validation, to filter different algorithms according to their performance, then it is not a valid test. It seems a bit trivial to me and I wonder what needs to be formal about this. The same is true for Kaggle competitions or p-values in research, it is all susceptible to confirmation bias. The way to solve it is re-test it with a fourth test or stricter standards. (For selecting algorithms you could save data by selecting the algorithms along with the hyper-parameter tuning) | |
| Oct 14, 2021 at 19:45 | comment | added | Sanyo Mn | I know the difference between validation and test set, please check my reply to @astel above. | |
| Oct 14, 2021 at 17:25 | history | answered | Sextus Empiricus | CC BY-SA 4.0 |