In Sutton and Barto's book, they talk about how state value functions can be estimated iteratively. First, we initialize the values of all states to arbitrary values. Then, we improve our estimations by applying the Bellman's equation multiple times.
My question is: if the dynamics of an MDP are known, why cant we work backwards to find the exact value functions? In other words, why can't we start from the terminal states and go to the starting states. Since we know that the value of the terminal states is zero, the value of the states preceding the terminal states would only depend on the immediate reward, which is known.