Temporal Difference Update Rule in Sutton and Barto

Question

In the Temporal Difference update rule that is given in Sutton and Barto section 1.4, it suggests back up of the values only after greedy moves are made. Is it not important to make updates even when exploratory moves are made?

If we do not make updates after exploratory moves and actually bump into a good exploratory move, how would we leverage the same and enforce the agent to not behave greedily?

Here is the formula for reference. $V(s) \gets V(s) +\alpha \times (V(s^f)-V(s))$

The text from the book is as follows.

To do this, we "back up" the value of the state after each greedy move to the state before the move, precisely, the current value of the earlier state is adjusted to be closer to the value of the later state. This can be done by moving the earlier state's value a fraction of the way toward the value of the later state. If we let s denote the state before the greedy move, and $s^f$ the state after, then the update to the estimated value of s, denoted $V(s)$, can be written as above.

Answer 1

If you want to learn the value of acting optimally, you cannot backup values from exploratory moves. Otherwise you end up including the value estimates of your explorations. This is discussed in more detail when covering on-policy vs off-policy learning later in the book.

The reason exploring still works is that you will end up in new state (or state-action pairs as you will see later in the book). You can backup greedy actions to this new state when you take the next move again, and update the new exploratory state's value. And if that newly explored state ends up having higher value than the original choice after that backup, it will be chosen greedily next time around.