2
$\begingroup$

If I can get N unbiased samples $x$ from $p(x)$ how can I approximate the Differential entropy:

$$H(X) = -\displaystyle\int_{x} p(x)\log p(x) dx$$

I'm not very knowledgeable in statistics so I'm not sure whether or not this is possible without making assumptions about the form of $p(x)$.

I forgot to mention that $X$ is a continuous random variable.

EDIT: Changed to Differential entropy thanks to correction from @gunes

Improve this question
$\endgroup$

2 Answers 2

Reset to default
2
$\begingroup$

First of all, for continuous variables, we actually calculate differential entropy, which is denoted as $h(X)$ in general.

When you have samples, you can construct a histogram. There may be various approaches after then. You can fit a parametric distribution if its shape is tameable, which means you need to make some assumptions of the form of $p(x)$, obtain parameters and perform the usual integration. Or, you can estimate $p(x)$ via a non-parametric method like KDE, and after that perform numeric integration to find $h(X)$.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Is there any parametric method in which I can compute the entropy or integral in closed form? Maybe Gaussian process? $\endgroup$
    – piccolo
    Commented Jan 19, 2019 at 20:21
  • $\begingroup$ If the shape is familiar, i.e. resembles well known distributions, choose one and estimate density parameters via MLE. Or, use Gaussian Mixture Models, which assumes the data is generated from a mixture of Gaussians, and you need to decide on the number of components. In either way, plot the fitted $p(x)$ over your sample histogram and convince yourself that it is a good fit. $\endgroup$
    – gunes
    Commented Jan 19, 2019 at 20:29
2
$\begingroup$

If you observe $x_1,\ldots,x_n$ as realisations from $p(x)$, the average $$\frac{-1}{n}\sum_{i=1}^n\log x_i$$is an unbiased and converging estimator of $H$.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I know what you're saying works in the limit of n to $\inf$ but I think assuming the distribution is smooth and continuous works better in the finite data regime. $\endgroup$
    – piccolo
    Commented Jan 20, 2019 at 15:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.