Why is having low variance important in offline policy evaluation of reinforcement learning?

Question

Intuitively, I understand that having an unbiased estimate of a policy is important because being biased just means that our estimate is distant from the truth value.

However, I don't understand clearly why having lower variance is important. Is that because, in offline policy evaluation, we can have only 'one' estimate with a stream of data, we don't know if it is because of variance or bias when our estimate is far from the truth value? Basically, variance acts like bias.

Also, if that is the case, why having variance is preferable to having bias?

Thank you in advance!

Answer 1

Often, we may find a good policy via gradient ascent wrt our estimate of the policy performance. If the variance is high, then the policy may converge slower, or perhaps not at all.

It's not always the case that variance is preferable to bias (for example, we might be willing to accept a small amount of bias if it substantially reduces variance), but being unbiased gives you the guarantee that if you do enough rollouts of your policy, you will converge on a good estimate and be able to improve your policy.