# Conceptual questions on Entropy and estimation

Learning Informative Statistics: A Nonparametric Approach paper presents an approach to parameter estimation by entropy minimization. There are other related works "Minimum-entropy estimation in semi-parametric models" download link ( http://dl.acm.org/citation.cfm?id=1195853). The rationale provided is that minimization of error entropy is equivalent to maximization of likelihood. I am new to this area and find it hard to understand the intuition behind why entropy of the error minimization will yield the parameters. What happens when entropy is minimized?

1. What happens when Shannon Entropy is maximized? Entropy (Shannon's) is the uncertainty = average information or uncertainty (unsure).

2. And what happens when entropy is minimized and

3. What is the meaning of minimizing entropy of error?

Entropy can be thought of as a non-parametric statistic about the spread of a distribution. Consider the follow unitless definition of entropy.

$$-\sum_n p_n\log_n p_n$$

If we define $0\log 0 = 0$, then the image of the function is $[0, 1]\forall n<\infty$.

Suppose that we have the uniform discrete distribution, with the pmf given by: $$1\over n$$

The value of our function for this distribution is 1 for all values of n.

Now consider the multinomial distribution given by: $$[ 0.01, 0.01, 0.01, 0.01, 0.91, 0.01, 0.01, 0.01, 0.01, 0.01]$$

The value of our function for this distribution is around 0.22.

Consider the multinomial given by: $$[ 0.001, 0.001, 0.001, 0.001, 0.001, 0.991, 0.001, 0.001, 0.001, 0.001]$$

The value of our function for this distribution is around 0.03.

So the more "concentrated" our distribution is, the more we know about the location of our realizations. It's easy to see by the definition of our function that it is invariant to translations of the location, or clustering of the high probability regions.

One naggle with entropy and related information theory measures, is that the unit is defined by the base of the logarithm. $e$ is nats, $2$ is bits. This means that, without normalization, the support of the distribution influences the measure.