Timeline for Validation Error less than training error?
Current License: CC BY-SA 3.0
13 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jul 3, 2021 at 1:57 | comment | added | Franva |
This is exactly my case~! Simple and clear~! thanks. I use FastAI 2.4 and when training I set dropout=0.5. Now it explains everything. Finally I can relax, gosh
|
|
| Jun 24, 2020 at 21:21 | comment | added | rocksNwaves | To expand on this answer even more, Often training loss is calculated in the middle of an epoch, whereas validation loss is calculated after the epoch is over. This could also cause a shift. pyimagesearch.com/2019/10/14/…. | |
| Oct 18, 2019 at 4:54 | comment | added | Cliff AB | Unfortunately, the main premise seems flawed. There's no reason why deactivating drop out would smooth over a function. | |
| Sep 8, 2019 at 22:46 | comment | added | Kanmani | If its the dropout that is really the culprit, is the resulting model really viable ? Is it advisable to reduce the dropout such that this phenomenon disappears ? | |
| Dec 13, 2018 at 21:07 | comment | added | Josiah Yoder | @MiloMinderbinder That's an interesting observation. No, I haven't figured anything else out on this yet. | |
| Oct 18, 2018 at 9:30 | comment | added | figs_and_nuts | @JosiahYoder - have you anything more to share on this? I have 1650 input features. when i keep the network small (1650, 50, 1) dropout or no dropout, the training error in the initial epochs is higher than validation error. When i use large networks (1650, 1200, 800, 100 ..... around 10 layers of 100 with selu activation), the weird pattern of higher validation accuracy is somewhat mitigated. | |
| Aug 9, 2018 at 17:34 | comment | added | André Christoffer Andersen | Fits with my case. Using lots of dropout. | |
| Jul 27, 2018 at 18:43 | comment | added | Josiah Yoder | I removed my dropout layer, but still see the validation loss lower than the training loss initially! (I am not specifying any regularization on the layers, either!) | |
| Jun 30, 2017 at 11:40 | comment | added | Simanas | Yes this should be marked as correct answer indeed. | |
| Mar 6, 2017 at 15:12 | comment | added | rajb245 | What a simple, sensible answer! | |
| Apr 6, 2016 at 14:17 | review | Late answers | |||
| Apr 6, 2016 at 14:20 | |||||
| Apr 6, 2016 at 14:03 | review | First posts | |||
| Apr 6, 2016 at 14:05 | |||||
| Apr 6, 2016 at 14:02 | history | answered | D-K | CC BY-SA 3.0 |