I'm studying rules of inference for updating from a prior probability distribution to a posterior. One method for doing this is by maximising entropy, subject to constraints. I'm reading papers like this: https://arxiv.org/pdf/physics/0608185.pdf

If $q$ is my prior distribution, and $p$ is my posterior, the relative entropy (Kullback-Leibler Divergence) between $p$ and $q$ is:

$$ D(p||q)=\int p(x)\log\frac{p(x)}{q(x)} \ dx $$

This is close to a metric on a space of probability measures, but it is not symmetric and the triangle inequality fails -- so it's not actually a metric. But it still measures something like the difference between distributions. In particular, in information theory, it's said to quantify the information gain by updating from $q$ to $p$.

Maximum entropy methods suggest maximising the relative entropy between $q$ and $p$, subject to constraints on $p$ (e.g. moment constraints or constraints given by observed data). But surely we want to minimise the relative entropy? When we update, we want to change our distribution as little as possible, while satisfying the constraints. This is why I find the 'maximimum relative entropy' terminology confusing.

Am I missing something that explains why it makes sense to talk about maximising relative entropy? Or did this terminology just derive from maximum (non-relative) entropy methods, where it does make sense to maximise entropy, when choosing priors?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.