In the Temporal Difference update rule that is given in Sutton and Barto section 1.4, it suggests back up of the values only after greedy moves are made. Is it not important to make updates even when exploratory moves are made?
If we do not make updates after exploratory moves and actually bump into a good exploratory move, how would we leverage the same and enforce the agent to not behave greedily?
Here is the formula for reference. $V(s) \gets V(s) +\alpha \times (V(s^f)-V(s))$
The text from the book is as follows.
To do this, we "back up" the value of the state after each greedy move to the state before the move, precisely, the current value of the earlier state is adjusted to be closer to the value of the later state. This can be done by moving the earlier state's value a fraction of the way toward the value of the later state. If we let s denote the state before the greedy move, and $s^f$ the state after, then the update to the estimated value of s, denoted $V(s)$, can be written as above.