Timeline for Is Deep-Q Learning inherently unstable
Current License: CC BY-SA 4.0
5 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jun 15, 2018 at 16:17 | vote | accept | enumaris | ||
| Jun 13, 2018 at 9:11 | comment | added | Neil Slater | I have seen more than one question regarding emulating DQN's success on Atari games, where the results have been unstable, and fixes include adding a target network (delayed copy of learning network) plus usually adjusting hyper-parameters for batch size and size of delay between learning network and target network. The behaviour when these are wrong can be similar to when learning rate is set too high. I would say this backs up this answer that DQN is inherently unstable, but still usable with a little patience thanks to the additions. | |
| Jun 13, 2018 at 3:41 | comment | added | shimao | @enumaris yes, it can be unstable in that sense, but I also meant unstable as in the performance of the policy wildly varies from epoch to epoch, the loss is all over the place, etc. Unstable in this sense does not prevent the network from eventually performing well. I believe diverging value estimates is usually solved using those tricks mentioned in my answer. | |
| Jun 13, 2018 at 3:35 | comment | added | enumaris | I've seen some sources talking about training instability, but they seem to focus on overfitting. By unstable, I mean that the value estimates can diverge as training occurs instead of converging to some value. Is Deep-Q Learning theoretically unstable in this sense? | |
| Jun 13, 2018 at 3:22 | history | answered | shimao | CC BY-SA 4.0 |