# Differential Entropy Estimation

Hello everyone, I am estimating differential entropy from a continuous distribution of a property of materials. The distributions on the picture are Gaussian Mixtures. The underlying Gaussians have some standard deviation which reaches also into the negative part of the x-axis, if the mean is close to zero. This produces confusion:

I want to express the amount of information this distribution contains via entropy. I know, that all of the properties are strictly positive, so the intuitive way is to truncate the distribution, in such a way that the part of the support that is negative is truncated from the distribution. But:

### Entropy is invariant to translation

Due to the fact that differential entropy is invariant to translation, and the amount of information is basically saying "how wide is the distribution", then I could just translate the distribution to the right and compute the entropy then (which would be the same as before the translation anyway, but at least theoretically valid) but this would no more mirror the prior knowledge of 'I know that properties are strictly positive.'

# The question:

What is the correct approach? Computing the differential entropy from the whole distribution also containing the negative part of the support or simply limit the computation of entropy to the positive part (i.e. truncation)?

# EDIT:

### As mentioned in the accepted answer, the approach of truncating the distribution is not solid, here is why:

• The general intuition behind entropy and measurements is that after each measurement, the entropy of the distribution should either lower or stay the same. It should not increase, that would mean loss of information in some sense.
• In the first picture, it can be seen that the distribution truncated at 0 has lower entropy. This makes sense as it has smaller footprint.
• In the second picture a measurement has been made. The truncated distribution has higher entropy after measurement (H(X) = 8.29292) than before the measurement (H(X) = 8.01642) which is incorrect.

# Conclusion

The truncation is incorrect approach and brings obscurity and haziness into the measurement.  • If you use a mixture of Gaussians, then your support will always be $\mathbb{R}$, i.e. negative and positive values. If you can change the underlying distribution from a mixture of Gaussians to something else, then consider a mixture of distributions with positive support, such as the truncated normal distribution or the chi-squared distribution. Aug 24, 2021 at 12:11
• I did not do a good job with using the word support, in retrospect. What I meant is, even though the support of Gaussians in general is $\mathbb{R}$, the valid support that makes sense in real life is $\mathbb{R}^+$ (strictly positive numbers) e.g. material density cannot be 0 or a negative number. Aug 24, 2021 at 12:19
• Please, could you upvote my question, so I can at leas upvote the answer, because it is good? Thanks. Aug 24, 2021 at 12:20