Timeline for Why do we need the score function in reinforcement learning?
Current License: CC BY-SA 3.0
10 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| May 22, 2018 at 19:02 | comment | added | Marcus | Regarding the difference between the policy optimization and value iteration methods. For me it dawned that policy optimization is like directly parameterizing to the actions with the loss/objective function, while value iteration more parameterises to the value of each state. It is like either parameterize the outcome of what each movement of the N64 controllers does directly while playing Mario Kart, or in the value iteration case parameterize towards the value of each game frame for the objective function. | |
| May 22, 2018 at 18:53 | comment | added | Marcus | I confused the return Gt which is the discouted sum of all the folowing rewards in that episode with R that is the retrun for that state, and the Q-value which is the return of the optimal action under the optimal policy. | |
| Apr 13, 2018 at 22:24 | comment | added | Neil Slater | The return is the discounted sum of rewards received since time $t$. It might correspond to (the difference in) score in a game. | |
| Apr 13, 2018 at 19:45 | comment | added | Marcus | Thanks again :) And Return Gt is just the score shown from example an atari game? trying to understand it from the basic level and up to the the theory, such as with value iteration and the basic gridworld example. Probably it seems the next thing to do is, look into Maximum likelihood estimation to understand this further. | |
| Apr 13, 2018 at 11:29 | history | edited | Neil Slater | CC BY-SA 3.0 |
added 16 characters in body
|
| Apr 13, 2018 at 11:26 | comment | added | Neil Slater | The simplest sampled policy gradient is the return $G_t$ times discounting factor $\gamma^t$ times the score function or $\gamma^t G_t \nabla_{\theta} ln(\pi(A_t|S_t, \theta))$. You can measure $G_t$ from an episode. The score function is the hard part, as you need to calculate it depending on your parametric function. If you have a specific policy function in mind (e.g. neural network with softmax action choice) and don't know how to do it for that, then I suggest ask a separate question. | |
| Apr 13, 2018 at 10:40 | comment | added | Marcus | Thank you! It explains more, I will look more into how the gradient is sampled to make the policy accent possible. Is the information sampled for the direct policy refinement similar to say a one step TD look ahead, or how do you obtain the gradient for the accent ? | |
| Apr 13, 2018 at 10:33 | vote | accept | Marcus | ||
| Apr 12, 2018 at 22:42 | history | edited | Neil Slater | CC BY-SA 3.0 |
added 951 characters in body
|
| Apr 12, 2018 at 22:24 | history | answered | Neil Slater | CC BY-SA 3.0 |