Timeline for Is it valid to change the model after seeing the results of test data?
Current License: CC BY-SA 4.0
17 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Aug 6, 2020 at 0:55 | vote | accept | Amir Hooshang | ||
| Jul 17, 2020 at 18:59 | answer | added | Newcomer | timeline score: 1 | |
| Jul 12, 2020 at 21:47 | history | edited | Amir Hooshang | CC BY-SA 4.0 |
deleted 5 characters in body
|
| Jul 12, 2020 at 20:40 | history | edited | Amir Hooshang | CC BY-SA 4.0 |
deleted 13 characters in body
|
| Jul 12, 2020 at 19:47 | comment | added | Peteris | @MostafaGhadimi evaluating it on the same test set afterwards yields some numbers that are almost worthless and can not be used to provide a reasonable evaluation estimete. If you want to know whether the changed model is appropriate or not, you would need new data, as you've made the previous test set useless for obtaining a realistic evaluation - you can only use it to get an "optimistic" relation that's likely to be "better" than the actual model performance by some unknown, unknowable amount. | |
| Jul 12, 2020 at 9:00 | history | tweeted | twitter.com/StackStats/status/1282238289114804224 | ||
| Jul 12, 2020 at 6:50 | history | became hot network question | |||
| Jul 12, 2020 at 1:24 | answer | added | Skander H. | timeline score: 11 | |
| Jul 12, 2020 at 0:43 | answer | added | Ben | timeline score: 21 | |
| Jul 11, 2020 at 23:59 | answer | added | Demetri Pananos | timeline score: 8 | |
| Jul 11, 2020 at 23:53 | comment | added | Amir Hooshang | @DemetriPananos by evaluating it with the same test set. | |
| Jul 11, 2020 at 23:52 | comment | added | Demetri Pananos | And so after changing the model, how would I determine in that model is appropriate or not? | |
| Jul 11, 2020 at 23:49 | comment | added | Amir Hooshang | @DemetriPananos yeah! Exactly. After fitting with training data then evaluate it with test data and finally change the model. | |
| Jul 11, 2020 at 23:47 | comment | added | Demetri Pananos | So you would for example fit a linear model, say “meh I don’t like this”, change to a different model, and then evaluate on the test set? Have I got that right? | |
| Jul 11, 2020 at 23:22 | comment | added | Amir Hooshang | @DemetriPananos the difference is where in first problem (link) it tunes the hyper-parameters that would destroy generalization. (Not overfitting in every situation) but in the latter problem, it uses completely different model, such as polynomial rather than linear. | |
| Jul 11, 2020 at 23:05 | comment | added | Demetri Pananos | I’m not sure how this is different. Could you provide some more detail? | |
| Jul 11, 2020 at 22:48 | history | asked | Amir Hooshang | CC BY-SA 4.0 |