In reinforcement learning, policy improvement is a part of an algorithm called policy iteration, which attempts to find approximate solutions to the Bellman optimality equations. Page-84, 85 in Sutton and Barto's book on RL mentions the following theorem:
Policy Improvement Theorem
Given two deterministic policies and :
RHS of inequality : the agent acts according to policy in the current state, and for all subsequent states acts according to policy
LHS of inequality : the agent acts according to policy starting from the current state.
In other words, is an improvement over .
I have a difficulty in understanding the proof. This is discussed below:
I am stuck here. The q-function is evaluated over the policy . That being the case, how is the expectation over the policy ?
My guess is the following: in the proof given in Sutton and Barto, the expectation is unrolled in time. At each time step, the agent follows the policy for that particular time step, and then follows from then on. In the limit of this process, the policy transforms from to . As long as the expression for the return inside the expectation is finite, the governing policy should be ; only in the limit of this process does the governing policy transform to .