Timeline for In MDPs with deterministic actions, should I use Q-learning or TD(0)?
Current License: CC BY-SA 3.0
4 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Apr 22, 2018 at 9:02 | comment | added | Dennis Soemers | As for applying Q-learning straight up in such games, that often doesn't work too well because Q-learning is an algorithm for single-agent problems, not for multi-agent problems. It does not inherently deal well with the whole minimax structure in games, where there are opponents selecting actions to minimize your value. Different forms of Reinforcement Learning (combined with tree search) have been used though, see the AlphaGo papers for example. | |
| Apr 22, 2018 at 9:00 | comment | added | Dennis Soemers | @czxttkl I never read the TD-gammon paper in detail, but from a very brief skim I get the impression that they do in fact just use TD, not Sarsa, so they just learn state-values. In order to make moves, they do indeed exploit the fact that they have knowledge of the game rules / a forward model: they simply generate all successor states $s'$, find all the $V(s')$ values for those states, and then select the move leading to the best successor. | |
| Apr 22, 2018 at 2:11 | comment | added | czxttkl | Thanks! Based on your answer, I think the work so-called"TD-gammon" is actually based on SARSA, which estimates $Q(s,a)$ for the current policy but also performs self-play so that the Q-values are constantly changing. Since we have a perfect forward model for back-gammon, $Q(s,a)$ actually becomes a state value $V(s')$, where $s'$ is the next state after $(s,a)$ . Am I right? Is there any reason that we don't see people applying Q-learning on games like Go and Back-Gammon? | |
| Apr 21, 2018 at 19:46 | history | answered | Dennis Soemers | CC BY-SA 3.0 |