Timeline for test accuracy is so much lower than validation accuracy by 6~10%. What could be the reason?
Current License: CC BY-SA 3.0
7 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Oct 25, 2017 at 8:53 | vote | accept | chahrazad | ||
| Oct 25, 2017 at 8:53 | |||||
| Oct 25, 2017 at 8:53 | comment | added | chahrazad | just for clarification the val set was used along training at each epoch to monitor performance out of sample, best performing model is saved, while training continues for a chosen total epochs. @StephanKolassa the test set is only for testing generalization, no further modification is done to the selected model. | |
| Oct 25, 2017 at 7:48 | comment | added | Stephan Kolassa | What @Scortchi says. Problems arise if you use your results on the "test set" to modify your model - because then the "test set" effectively becomes part of the validation set, which I argue is in turn actually part of the training set. | |
| Oct 25, 2017 at 7:46 | comment | added | Scortchi♦ | In general an estimate of the performance of the best-performing model on a validation set will be something of an over-estimate, the more so the more models you compare. That's why you have a test set: to get an unbiased estimate of a model's performance. | |
| Oct 25, 2017 at 7:43 | answer | added | Stephan Kolassa | timeline score: 6 | |
| Oct 25, 2017 at 7:41 | comment | added | Krrr | possibly the test data happens to be different than train/val. Repeat the procedure with another random sampling to make sure that is not an issue. | |
| Oct 25, 2017 at 7:17 | history | asked | chahrazad | CC BY-SA 3.0 |