One of the primary differences I've seen stated between Q-learning, as a model-free algorithm, and Value Iteration, as a model-based algorithm is that Value-Iteration requires a "model of the environment" in that it incorporates knowledge of how to transition from one state to another (which I've understood as a policy over actions), and knowledge of the rewards received at each state. I've also seen it stated that
"Value iteration works fine, but it has two weaknesses: first, it can take a long time to converge in some situations, even when the underlying policy is not changing"
Which has left me quite confused.
While I can see that it needs a given set of transition probabilities, I don't see anything in its update equation as to why it requires knowledge of rewards before hand and why it cannot be trained in an online way in the same way that Q-learning can? The next thing is that I'm not sure why the policy would ever change in value iteration? I thought the whole idea was to "predict" how this particular policy would perform using value iteration to find the optimal value function for this policy, and then "control" this behaviour using policy iteration - "one would adapt the previous policy slightly, perform value iteration, find a new value function, compare it to the previous value function and check if it had converged.
Is this an incorrect interpretation?
The equation given for value iteration is displayed here:
$ U(s) = R(s) + γmax_a\sum{T(s, a, s')U(s')} $
$ Q(s, a) = R(s, a) + γmax_a Q(s', a')$