2
$\begingroup$

Learning Informative Statistics: A Nonparametric Approach paper presents an approach to parameter estimation by entropy minimization. There are other related works "Minimum-entropy estimation in semi-parametric models" download link ( http://dl.acm.org/citation.cfm?id=1195853). The rationale provided is that minimization of error entropy is equivalent to maximization of likelihood. I am new to this area and find it hard to understand the intuition behind why entropy of the error minimization will yield the parameters. What happens when entropy is minimized?

  1. What happens when Shannon Entropy is maximized? Entropy (Shannon's) is the uncertainty = average information or uncertainty (unsure).

  2. And what happens when entropy is minimized and

  3. What is the meaning of minimizing entropy of error?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Entropy can be thought of as a non-parametric statistic about the spread of a distribution. Consider the follow unitless definition of entropy.

$$-\sum_n p_n\log_n p_n$$

If we define $0\log 0 = 0$, then the image of the function is $[0, 1]\forall n<\infty$.

Suppose that we have the uniform discrete distribution, with the pmf given by: $$1\over n$$

The value of our function for this distribution is 1 for all values of n.

Now consider the multinomial distribution given by: $$[ 0.01, 0.01, 0.01, 0.01, 0.91, 0.01, 0.01, 0.01, 0.01, 0.01]$$

The value of our function for this distribution is around 0.22.

Consider the multinomial given by: $$[ 0.001, 0.001, 0.001, 0.001, 0.001, 0.991, 0.001, 0.001, 0.001, 0.001]$$

The value of our function for this distribution is around 0.03.

So the more "concentrated" our distribution is, the more we know about the location of our realizations. It's easy to see by the definition of our function that it is invariant to translations of the location, or clustering of the high probability regions.

Regarding your questions, in the context of the linked paper.

  1. The uniform distribution has the largest entropy of any distribution. There is no information regarding the location of realized samples from this distribution because all locations are equally probable.
  2. If we imagine a dirac delta like distribution, where the mean equals the mode equals the max, equals the min, then the entropy is minimized. Minimization of the entropy of the distribution of residuals can be worded as gaining the most information about the location of the residuals. If all of the regression residuals are 1.5 then the entropy is minimized.
  3. See 2

One naggle with entropy and related information theory measures, is that the unit is defined by the base of the logarithm. $e$ is nats, $2$ is bits. This means that, without normalization, the support of the distribution influences the measure.

Improve this answer
$\endgroup$
4
  • $\begingroup$ The values come the defined equation for unitless entropy. Entropy tells you about the uncertainty of the random variable. The paper discusses entropy minimization of regression residuals. The value 1.5 was randomly chosen, but any constant value for the residuals will be the minimum of the entropy. The flatter the probability mass, the higher the entropy. Two headed coin has an entropy of 0. Fair coin has an entropy of 1. $\endgroup$
    – Jessica Collins
    Commented Jul 26, 2014 at 21:28
  • $\begingroup$ Actually, the opposite. We have more information about a random variable when it's entropy is smaller. Minimizing the entropy of the distribution of the residuals from the regression is like maximizing homoscedasticity. $\endgroup$
    – Jessica Collins
    Commented Jul 26, 2014 at 21:39
  • $\begingroup$ The principle of maximum entropy is unrelated to this, which is where some confusion may lay. Wikipedia has a good article on homoscedasticity. $\endgroup$
    – Jessica Collins
    Commented Jul 26, 2014 at 21:50
  • $\begingroup$ Bigger entropy means a flatter distribution. For a random variate, this means we have less information. If, on the other hand, we have several distributions that fit a set of data, then we should always prefer the one with the largest entropy because it best reflects our uninformedness about the situation. I wouldn't go so far as you said about minimizing entropy of error distributions. I'd stick with maximizing homoscedasticity of the model. $\endgroup$
    – Jessica Collins
    Commented Jul 26, 2014 at 22:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.