Why does this value iteration example converge in a finite number of steps? 
Could someone help me understand why value iteration converges for state C in 3 steps, and in 4 steps for everything else? Why not infinite?
 A: Note that the question points out that one needn't solve value iteration (VI) explicitly. It's hinting that from any state, there is a finite maximum number of transitions until the absorbing state. 
Loosely, this should help you see that there are only so many states value can "trickle back" from to either C or any other state. Running VI explicitly will confirm this.
A: When you have a finite acyclic graph like this, value iteration converges finitely. At the beginning, you know the value of state F (it's given). After one iteration, states that only feed into F then have their value correct (E,D). Another iteration, and states that only feed into (F,E,D) are locked in (B). 3rd iteration, then states that feed into (F,E,D,B) only are locked in (C). Finally, the 4th iteration locks in A.
So when the second question in your screenshot states that it takes 4 steps for all states to converge, it doesn't mean that all states other than C take 4 steps, rather it means that they take a maximum of 4 steps. 
