In Sutton and Barto's book: Reinforcement Learning: An Introduction, they write
Value functions define a partial ordering over policies.
My questions is why the value function is not a total order?
I would have thought: $\pi_1 \geq \pi_2$ if and only if $V_{\pi_1} (s_0) \geq V_{\pi_2} (s_0)$, where $s_0$ is the initial state distribution.
The follow sentence says:
A policy $\pi$ is defined to be better than or equal to a policy $\pi'$ if its expected return is greater than or equal to that of $\pi'$ for all states. In other words, $\pi \geq \pi'$ if and only if $V^\pi(s) \geq V^{\pi'}(s)$ for all $s \in \mathcal{S}$.
Of course, it's possible to define an ordering with respect to some state distribution, as you've suggested, but the state independent version is probably more useful.
