@Alberto It is admittedly an unclear setup: I am trying to learn the distribution $p$, but the $x_i$s are *not drawn from p*. They are drawn from some (unknown) biased distribution. However, what I *do* have are the likelihood values $p(x_i)$ for each data point. I.e., I have some information about the likelihood I would have seen each $x_i$ under the target distribution. If it makes it any easier, lets at least assume that we know the $x_i$s are drawn from some distribution with the same support as $p$.

I like this example, but feel the final paragraph doesn't quite answer the question. Clearly, once we fix the amount of data collected, value iteration will converge for any learned transition function. I'm interested in the case when an agent can go on collecting data forever. Under what conditions will the agent stabilize to one fixed policy (even if it is the wrong one)? It seems like with the specific case of the left/right example, the answer is that it will fluctuate forever between two policies. Is this true in general? Or is there a subset of POMDPs where this is not the case?