Comparing divergence from uniform distributions with differing supports (discrete) Imagine we have a potentially biased coin and a potentially biased six-sided die and we want to know which is more biased than the other.
Firstly, is this a reasonable goal? Could it make sense to say, for example, that the coin is more biased than the die?
Secondly, assuming the goal is reasonable, can we use the KL-divergence from the natural uniform (unbiased) distribution to the observed distribution as a measure of bias in such a setup? If not, what might we use instead?
Lastly, if using KL-divergence is reasonable, can we obtain any meaning from the degree of difference between the divergences? For example, if the coin diverges by 2 and the die diverges by 1, can we say the coin is twice as biased as the die?
The coin is to be compared with the uniform distribution over two outcomes while the die is to be compared with a uniform distribution over six outcomes. Clearly, if both discrete random variables were over the same number of possible outcomes, a direct comparison of the KL-divergences would be reasonable. But does this remain true when the number of the outcomes differs?
 A: One method that comes to mind is perplexity, which is a common idea in information theory and NLP. Essentially it is defined as
$$
\text{Perplexity} = 2^{-\sum_{i}p_i\log_2 p_i}
$$
where the exponent can readily be interpreted as the entropy of your probability distribution in bits. The general idea is that if your distribution were encoded as a fair n-sided die, how many faces would it have?
For instance, a fair 6-sided die has distribution $\{\frac{1}{6},\frac{1}{6},\frac{1}{6},\frac{1}{6},\frac{1}{6},\frac{1}{6}\}$ so it has perplexity
$$
\text{Perplexity} = 2^{-\sum_{i=1}^{6}\frac{1}{6}\log_2 \frac{1}{6}} = 2^{\log_2 6} = 6
$$
as we would expect. Now let's consider a die which has distribution $\{\frac{1}{2},\frac{1}{2},0,0,0,0\}$. This has perplexity 2 (note that $0\log 0 = 0$ for us) so is equivalent to a 2-sided die, or an unbiased coin. Try this on a few other examples and you'll find it's a really nice measure of how far a discrete distribution is away from uniform, what you called 'biased'.
Note that you don't have to use the base 2, but it is the most standard form, followed by $e$.
