2
$\begingroup$

Differential Entropy Estimation

Hello everyone, I am estimating differential entropy from a continuous distribution of a property of materials. The distributions on the picture are Gaussian Mixtures. The underlying Gaussians have some standard deviation which reaches also into the negative part of the x-axis, if the mean is close to zero. This produces confusion:

I want to express the amount of information this distribution contains via entropy. I know, that all of the properties are strictly positive, so the intuitive way is to truncate the distribution, in such a way that the part of the support that is negative is truncated from the distribution. But:

Entropy is invariant to translation

Due to the fact that differential entropy is invariant to translation, and the amount of information is basically saying "how wide is the distribution", then I could just translate the distribution to the right and compute the entropy then (which would be the same as before the translation anyway, but at least theoretically valid) but this would no more mirror the prior knowledge of 'I know that properties are strictly positive.'

The question:

What is the correct approach? Computing the differential entropy from the whole distribution also containing the negative part of the support or simply limit the computation of entropy to the positive part (i.e. truncation)?

EDIT:

As mentioned in the accepted answer, the approach of truncating the distribution is not solid, here is why:

  • The general intuition behind entropy and measurements is that after each measurement, the entropy of the distribution should either lower or stay the same. It should not increase, that would mean loss of information in some sense.
  • In the first picture, it can be seen that the distribution truncated at 0 has lower entropy. This makes sense as it has smaller footprint.
  • In the second picture a measurement has been made. The truncated distribution has higher entropy after measurement (H(X) = 8.29292) than before the measurement (H(X) = 8.01642) which is incorrect.

Conclusion

The truncation is incorrect approach and brings obscurity and haziness into the measurement.

image 1 image 2

Improve this question
$\endgroup$
3
  • $\begingroup$ If you use a mixture of Gaussians, then your support will always be $\mathbb{R}$, i.e. negative and positive values. If you can change the underlying distribution from a mixture of Gaussians to something else, then consider a mixture of distributions with positive support, such as the truncated normal distribution or the chi-squared distribution. $\endgroup$
    – mhdadk
    Aug 24, 2021 at 12:11
  • $\begingroup$ I did not do a good job with using the word support, in retrospect. What I meant is, even though the support of Gaussians in general is $\mathbb{R}$, the valid support that makes sense in real life is $\mathbb{R}^+$ (strictly positive numbers) e.g. material density cannot be 0 or a negative number. $\endgroup$
    – Andrej Kružliak
    Aug 24, 2021 at 12:19
  • $\begingroup$ Please, could you upvote my question, so I can at leas upvote the answer, because it is good? Thanks. $\endgroup$
    – Andrej Kružliak
    Aug 24, 2021 at 12:20

1 Answer 1

Reset to default
2
$\begingroup$

In general, the approach is not solid. If a sizeable fraction of your model leaks into the forbidden region, then the model does a poor job at describing the true probability distribution, and, by extension, any metric derived from that model would incur a certain error. Cropping the probability distribution at zero and re-scaling may or may not be cheating. If you have reasons to believe that the cropped and re-scaled model is a good model of the underlying probability distribution, then go ahead. If not then you may be introducing errors.

My suggestion is as follows. Do it both ways and compare the results. If the amount of leak is small and the results agree to a good precision, then use either of the results. If the leak is sizeable and/or the results disagree significantly, then you have to change your model to a model that is explicitly non-zero

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks, I have tried out some counter-examples and it looks like your answer is correct. $\endgroup$
    – Andrej Kružliak
    Aug 25, 2021 at 9:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.