While reading the book: Reinforcement Learning, An Introduction by Rich Sutton, I came across a doubt regarding afterstates and afterstates value functions and I am afraid I don't understand the concept well.
A conventional state-value function evaluates states in which the agent has the option of selecting an action, but the state-value function used in tic-tac-toe evaluates board positions after the agent has made its move. Let us call these afterstates, and value functions over these, afterstate value functions.
Now, I understand what afterstates are, but I am unable to understand, how to use these afterstate value functions to learn the action values?
A conventional action-value function would have to separately assess both pairs, whereas an afterstate value function would immediately assess both equally. Any learning about the position–move pair on the left would immediately transfer to the pair on the right.
How do we make sure that the learning is transferred from one state-action pair to the other provided both the state-action pair results in the same afterstate?