Differential Entropy Estimation
Hello everyone, I am estimating differential entropy from a continuous distribution of a property of materials. The distributions on the picture are Gaussian Mixtures. The underlying Gaussians have some standard deviation which reaches also into the negative part of the x-axis, if the mean is close to zero. This produces confusion:
I want to express the amount of information this distribution contains via entropy. I know, that all of the properties are strictly positive, so the intuitive way is to truncate the distribution, in such a way that the part of the support that is negative is truncated from the distribution. But:
Entropy is invariant to translation
Due to the fact that differential entropy is invariant to translation, and the amount of information is basically saying "how wide is the distribution", then I could just translate the distribution to the right and compute the entropy then (which would be the same as before the translation anyway, but at least theoretically valid) but this would no more mirror the prior knowledge of 'I know that properties are strictly positive.'
What is the correct approach? Computing the differential entropy from the whole distribution also containing the negative part of the support or simply limit the computation of entropy to the positive part (i.e. truncation)?
As mentioned in the accepted answer, the approach of truncating the distribution is not solid, here is why:
- The general intuition behind entropy and measurements is that after each measurement, the entropy of the distribution should either lower or stay the same. It should not increase, that would mean loss of information in some sense.
- In the first picture, it can be seen that the distribution truncated at 0 has lower entropy. This makes sense as it has smaller footprint.
- In the second picture a measurement has been made. The truncated distribution has higher entropy after measurement
(H(X) = 8.29292)than before the measurement
(H(X) = 8.01642)which is incorrect.
The truncation is incorrect approach and brings obscurity and haziness into the measurement.