To compare two KL-divergence scores, does the prior model have to be the same for both?

The KL-divergence compares a theoretical model $$p$$'s distribution with the empirical model $$q$$'s distribution, giving a score of $$0$$ if they, or their information contents, are identical.

Say we have two KL-divergences for two distinct models (that have the same units and context), therefore two pairs (4) distributions

1. If the first score $$KLD_1=0.3$$ tells us how far off model 1 ($$p_1$$) is in reality from theoretical model 1 ($$q_1$$),
2. and the second score $$KLD_2=0.4$$ quantifies how far off model 2 is in reality ($$p_2$$) from theoretical model 2 ($$q_2$$),

can I compare the values of $$KLD_1$$ and $$KLD_2$$ directly and conclude "model 1 outperformed model 2 because its score was lower, implying less divergence from its own theoretical prior" even though the scores were respectively measured based on two different models/priors? Or are two KLD scores only comparable if based on the same underlying theoretical model, i.e. $$(p_1 || \mathbf{q_1})$$ vs $$(p_2 || \mathbf{q_1})$$?

The question more abstractly,

1. KLD(red apples || green apples) vs KLD(red grapes || green grapes), or
2. KLD(red apples || mangos) vs KLD(red grapes || mangos)?

1 Answer

Yes, that is possible but chances are it is not the kind of performance you had in mind originally. Essentially, you are saying using model 1's predicted distribution to encode information about the "true" distribution 1 is more efficient than using model 2's predicted distribution to encode information about the "true" distribution 2.

This statement makes no claim about the predictive ability of either model. A model that predicts an ideal coin flip with odds of 50:50 will be right about as often as it is wrong and will have KLD = 0. Contrast this with a model that predicts that I will always eat a red apple that I bought. It will be right more often than it will be wrong (I do eat most of the apples that I buy), but sometimes I give the apple to my girlfriend instead, so the model doesn't match the "true" distribution and hence KLD > 0. Which of the two models makes better predictions?

Additionally, the models predict different things, so there is no reason to prefer one over the other; we can happily use both in tandem.