Conceptual questions on Entropy and estimation

Question

Learning Informative Statistics: A Nonparametric Approach paper presents an approach to parameter estimation by entropy minimization. There are other related works "Minimum-entropy estimation in semi-parametric models" download link ( http://dl.acm.org/citation.cfm?id=1195853). The rationale provided is that minimization of error entropy is equivalent to maximization of likelihood. I am new to this area and find it hard to understand the intuition behind why entropy of the error minimization will yield the parameters. What happens when entropy is minimized?

1. What happens when Shannon Entropy is maximized? Entropy (Shannon's) is the uncertainty = average information or uncertainty (unsure).

2. And what happens when entropy is minimized and

3. What is the meaning of minimizing entropy of error?

Answer 1

Entropy can be thought of as a non-parametric statistic about the spread of a distribution. Consider the follow unitless definition of entropy.

$$-\sum_n p_n\log_n p_n$$

If we define $0\log 0 = 0$, then the image of the function is $[0, 1]\forall n<\infty$.

Suppose that we have the uniform discrete distribution, with the pmf given by: $$1\over n$$

The value of our function for this distribution is 1 for all values of n.

Now consider the multinomial distribution given by: $$[ 0.01, 0.01, 0.01, 0.01, 0.91, 0.01, 0.01, 0.01, 0.01, 0.01]$$

The value of our function for this distribution is around 0.22.

Consider the multinomial given by: $$[ 0.001, 0.001, 0.001, 0.001, 0.001, 0.991, 0.001, 0.001, 0.001, 0.001]$$

The value of our function for this distribution is around 0.03.

So the more "concentrated" our distribution is, the more we know about the location of our realizations. It's easy to see by the definition of our function that it is invariant to translations of the location, or clustering of the high probability regions.

Regarding your questions, in the context of the linked paper.

1. The uniform distribution has the largest entropy of any distribution. There is no information regarding the location of realized samples from this distribution because all locations are equally probable.
2. If we imagine a dirac delta like distribution, where the mean equals the mode equals the max, equals the min, then the entropy is minimized. Minimization of the entropy of the distribution of residuals can be worded as gaining the most information about the location of the residuals. If all of the regression residuals are 1.5 then the entropy is minimized.
3. See 2

One naggle with entropy and related information theory measures, is that the unit is defined by the base of the logarithm. $e$ is nats, $2$ is bits. This means that, without normalization, the support of the distribution influences the measure.