Imagine we have a potentially biased coin and a potentially biased six-sided die and we want to know which is more biased than the other.
Firstly, is this a reasonable goal? Could it make sense to say, for example, that the coin is more biased than the die?
Secondly, assuming the goal is reasonable, can we use the KL-divergence from the natural uniform (unbiased) distribution to the observed distribution as a measure of bias in such a setup? If not, what might we use instead?
Lastly, if using KL-divergence is reasonable, can we obtain any meaning from the degree of difference between the divergences? For example, if the coin diverges by 2 and the die diverges by 1, can we say the coin is twice as biased as the die?
The coin is to be compared with the uniform distribution over two outcomes while the die is to be compared with a uniform distribution over six outcomes. Clearly, if both discrete random variables were over the same number of possible outcomes, a direct comparison of the KL-divergences would be reasonable. But does this remain true when the number of the outcomes differs?