What is the difference between off-policy and on-policy learning?

Question

Artificial intelligence website defines off-policy and on-policy learning as follows:

"An off-policy learner learns the value of the optimal policy independently of the agent's actions. Q-learning is an off-policy learner. An on-policy learner learns the value of the policy being carried out by the agent including the exploration steps."

I would like to ask your clarification regarding this, because they don't seem to make any difference to me. Both the definitions seem like they are identical. What I actually understood are the model-free and model-based learning, and I don't know if they have anything to do with the ones in question.

How is it possible that the optimal policy is learned independently of the agent's actions? Isn't the policy learned when the agent performs the actions?

Answer 1

First of all, there's no reason that an agent has to do the greedy action; Agents can explore or they can follow options. This is not what separates on-policy from off-policy learning.

The reason that Q-learning is off-policy is that it updates its Q-values using the Q-value of the next state $s'$ and the greedy action $a'$. In other words, it estimates the return (total discounted future reward) for state-action pairs assuming a greedy policy were followed despite the fact that it's not following a greedy policy.

The reason that SARSA is on-policy is that it updates its Q-values using the Q-value of the next state $s'$ and the current policy's action $a''$. It estimates the return for state-action pairs assuming the current policy continues to be followed.

The distinction disappears if the current policy is a greedy policy. However, such an agent would not be good since it never explores.

Have you looked at the book available for free online? Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. Second edition, MIT Press, Cambridge, MA, 2018.

Answer 2

First of all, what does policy (denoted by $\pi$) actually mean?
Policy specifies an action $a$, that is taken in a state $s$ (or more precisely, $\pi$ is a probability, that an action $a$ is taken in a state $s$).

Second, what types of learning do we have?

1. Evaluate $Q(s,a)$ function: predict sum of future discounted rewards, where $a$ is an action and $s$ is a state.
2. Find $\pi$ (actually, $\pi(a|s)$), that yields a maximum reward.

Back to the original question. On-policy and off-policy learning is only related to the first task: evaluating $Q(s,a)$.

The difference is this:
In on-policy learning, the $Q(s,a)$ function is learned from actions that we took using our current policy $\pi(a|s)$.
In off-policy learning, the $Q(s,a)$ function is learned from taking different actions (for example, random actions). We don't even need a policy at all!

This is the update function for the on-policy SARSA algorithm: $Q(s,a) \leftarrow Q(s,a)+\alpha(r+\gamma Q(s',a')-Q(s,a))$, where $a'$ is the action, that was taken according to policy $\pi$.

Compare it with the update function for the off-policy Q-learning algorithm: $Q(s,a) \leftarrow Q(s,a)+\alpha(r+\gamma \max_{a'}Q(s',a')-Q(s,a))$, where $a'$ are all actions, that were probed in state $s'$.

Answer 3

On-policy methods estimate the value of a policy while using it for control.

In off-policy methods, the policy used to generate behaviour, called the behaviour policy, may be unrelated to the policy that is evaluated and improved, called the estimation policy.

An advantage of this seperation is that the estimation policy may be deterministic (e.g. greedy), while the behaviour policy can continue to sample all possible actions.

For further details, see sections 5.4 and 5.6 of the book Reinforcement Learning: An Introduction by Barto and Sutton, first edition.

Answer 4

The difference between Off-policy and On-policy methods is that with the first you do not need to follow any specific policy, your agent could even behave randomly and despite this, off-policy methods can still find the optimal policy. On the other hand on-policy methods are dependent on the policy used. In the case of Q-Learning, which is off-policy, it will find the optimal policy independent of the policy used during exploration, however this is true only when you visit the different states enough times. You can find in the original paper by Watkins the actual proof that shows this very nice property of Q-Learning. There is however a trade-off and that is off-policy methods tend to be slower than on-policy methods. Here a link with other interesting summary of the properties of both types of methods

Answer 5

On-policy learning: The same (ϵ-greedy) policy that is evaluated and improved is also used to select actions. For eg. SARSA TD Learning Algorithm

Off-policy learning: The (greedy) policy that is evaluated and improved is different from the (ϵ-greedy) policy that is used to select actions. For eg. Q-Learning Algorithm

Answer 6

From the Sutton book: "The on-policy approach in the preceding section is actually a compromise—it learns action values not for the optimal policy, but for a near-optimal policy that still explores. A more straightforward approach is to use two policies, one that is learned about and that becomes the optimal policy, and one that is more exploratory and is used to generate behavior. The policy being learned about is called the target policy, and the policy used to generate behavior is called the behavior policy. In this case we say that learning is from data “o↵” the target policy, and the overall process is termed o↵-policy learning."

Answer 7

This is the recursive version of the Q-function (according to Bellman equation):

$$Q_\pi(s_t,a_t)=\mathbb{E}_{\,r_t,\,s_{t+1}\,\sim\,E}\left[r(s_t,a_t)+\gamma\,\mathbb{E}_{\,a_{t+1}\,\sim\,\pi}\left[Q_\pi(s_{t+1}, a_{t+1})\right]\right]$$

Notice that the outer expectation exists because the current reward and the next state are sampled ($\sim)$ from the environment ($E$). The inner expectation exists because the Q-value for the next state depends on the next action. If you your policy is deterministic, there is no inner expectation, our $a_{t+1}$ is a known value that depends only on the next state, let's call it $A(s_{t+1})$:

$$Q_{det}(s_t,a_t)=\mathbb{E}_{\,r_t,\,s_{t+1}\,\sim\,E}\left[r(s_t,a_t)+\gamma\,Q_{det}(s_{t+1}, A(s_{t+1})\right]$$

This means the Q-value depends only on the environment for deterministic policies.

The optimal policy is always deterministic (it always take the action that leads to higher expected reward) and Q-learning directly approximates the optimal policy. Therefore the Q-values of this greedy agent depends only on the environment.

Well, if the Q-values depends only on the environment, it doesn't matter how I explore the environment, that is, I can use an exploratory behaviour policy.

Answer 8

In the way I understood it, hope it helps :

On-policy learning updates the policy currently in use while off-policy learning updates a different policy using the data collected from a different policy.

• On-policy learning is a type of RL that updates the policy being used to take actions as the agent interacts with the environment. Specifically, the agent learns by following the current policy and then updates the policy based on the rewards received from those actions. (It is often used in situations where the agent's exploration of the environment is limited and the learning must be done with the current policy).

• Off-policy learning, updates a different policy than the one being used to take actions. This approach involves learning from the behavior of an "older" policy (or another one), while simultaneously interacting with the environment using a newer, improved policy (currently learned). (The main advantage of off-policy learning is that it allows for greater exploration of the environment, which can lead to better policies). Technically, the book is called buffer.

Example: an agent playing a game of chess. With on-policy learning, the agent would learn by playing the game using its current policy and then update its policy based on the rewards it receives (experience is sampled from the updated policy). But with the off-policy learning, the agent might study a chess book to learn new strategies, and then incorporate those strategies into its policy while still playing games using its original policy (experience is sampled from the "book" policy here).

I recommend to read this following article : https://medium.com/@sergey.levine/decisions-from-data-how-offline-reinforcement-learning-will-change-how-we-use-ml-24d98cb069b0

Answer 9

On-policy methods attempt to evaluate or improve the policy that is used to make decisions, whereas off-policy methods evaluate or improve a policy different from that used to generate the data. [1]

[1]. Reinforcement Learning: An Introduction. Second edition, in progress. Richard S. Sutton and Andrew G. Barto c 2014, 2015. A Bradford Book. The MIT Press.

Answer 10

I find this helpful: Michael Herrmann: On-Policy and Off-Policy Algorithm

Answer 11

An off-policy learner learns the value of the optimal policy independently of the agent's actions

should be clarified as the data from which off-policy learner learns is independent of the agent's current policy (the policy in the moment the learner update its Q table(or Q network))

I prefer to understand why Q learnig is off-policy alongside DQN. In DQN, we can use memory buffer, where we definitely update the policy using data generated from old policy, which meets the definition above.

As for q learning, what is confusing is that actually we don't use memory buffer(at least in sutton's book), the learning data is actually generated, or more precisely, influenced by our current policy! That is more like on-policy! But wait, that doesn't mean we can't use it in completely off-policy manner. Like Dqn, memory buffer is also applicable to q learning.

Back to Q-learning vs SARSA, it is not hard to see that SARSA can't fit memory buffer, we can't use old-policy-generating data to update our q table. Becanse we use the current policy to estimate the next state's value V(s'). So Neil's answer did tell the crucial point behind all I say. But you can go further to fit the definition better.