I was trying to understand the proof why policy improvement theorem can be applied on epsilon-greedy policy.
The proof starts with the mathematical definition -
I am confused on the very first line of the proof. In an MDP -
This equation is the Bellman expectation equation for Q(s,a), while V(s) and Q(s,a) follow the relation -
So how can we ever derive the first line of the proof ? What does the notation Q(s, π'(s)) even mean, since π is not a deterministic policy but a stochastic one (so it doesn't refer to a fixed action) ? Isn't the Q value only defined for a state action pair ?