Timeline for What is the difference between off-policy and on-policy learning?
Current License: CC BY-SA 4.0
15 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| S Dec 3, 2022 at 14:26 | history | suggested | GregarityNow | CC BY-SA 4.0 |
fixed grammar to improve readability
|
| Dec 3, 2022 at 11:25 | review | Suggested edits | |||
| S Dec 3, 2022 at 14:26 | |||||
| Apr 21, 2022 at 16:46 | comment | added | Albert Chen | to be more accurate, the difference is not about how Q is learned, it's about how the RETURN is approximated. | |
| Mar 4, 2022 at 19:09 | comment | added | Appliqué | @PavelKomarov Let's assume we use $\varepsilon$-greedy policy. Then in the on-policy formula $a'$ in $Q(s', a')$ stands for the $\varepsilon$-greedy action which is different from $\mathrm{argmax}_{a'} Q(s', a')$. | |
| Nov 15, 2021 at 17:12 | comment | added | Pavel Komarov | I don't really understand this distinction. Isn't $a' = \pi(a'|s')$ just equal to $a' = max_{a'} Q(s', a')$ in the off policy case? That's still a policy, just not one you need to store separately from the Q function. Does the distinction lie in the way we often keep a separate target Q in some flavors of RL that we only update with information from the working Q every $k$ iterations? | |
| Nov 11, 2021 at 18:26 | comment | added | Sarvagya Gupta | So can we say that algorithms like policy gradient are a mix of on and off-policy learning? Because there is this exploration-exploitation rate that tells the RL when to be greedy and when to explore | |
| Mar 16, 2021 at 21:49 | comment | added | chimbo | @alex If I understand correctly, a policy is a function of current state and environment, while taking random actions would not take current state/environment into account. I guess you could have a function that just outputs random actions no matter the input, but then whether that's an actual "policy" is debatable. | |
| Jun 8, 2020 at 3:43 | comment | added | alex | "In off-policy learning, the $Q(s,a)$ function is learned from taking different actions (for example, random actions). We even don't need a policy at all!" - How can you not have a policy? Isn't even taking random actions technically a policy? Also it would be helpful if you could illucidate the difference in the Q update between SARSA and Q-Learning that show what makes either on policy or off policy. | |
| S May 30, 2020 at 16:57 | history | suggested | Rasoul | CC BY-SA 4.0 |
language correction
|
| May 29, 2020 at 8:24 | review | Suggested edits | |||
| S May 30, 2020 at 16:57 | |||||
| Dec 12, 2018 at 8:33 | history | edited | Dmitry Mottl | CC BY-SA 4.0 |
deleted 4 characters in body
|
| Dec 11, 2018 at 16:41 | history | edited | Dmitry Mottl | CC BY-SA 4.0 |
added 4 characters in body
|
| Nov 13, 2018 at 21:05 | review | Late answers | |||
| Nov 13, 2018 at 21:09 | |||||
| Nov 13, 2018 at 20:55 | review | First posts | |||
| Nov 13, 2018 at 21:34 | |||||
| Nov 13, 2018 at 20:50 | history | answered | Dmitry Mottl | CC BY-SA 4.0 |