I have a bayesian model that gives me a sample of a distribution for some parameter x - a list of 100 continuous values like so: [1.32, 1.38, 1.44, ..., 1.28]. If I know that a sample is very disperse, I can request additional information to be inputted by hand - so I can ask "Is x lower than 1.34?

Additionally, for some parameters, there are only 1 or 2 possible values, and the model outputs 1 or 2 gaussians around those values (see the image below). If for all parameters I had a fixed number of possible values, I could easily compute the entropy, by treating them as categorical values. Unfortunately, that is not the case.

I could very easily make up some heuristic rule, but I'd like to know how to quantify how much entropy does a parameter contain. If I knew that, I would be able to easily minimize the amount of queries to the data entry team. My current solution is to run kernel density estimation, and for sum the information per kernel. How can I calculate entropy in a more reasonable way?

enter image description here


1 Answer 1


Your question, as I understand it, basically amounts to, "what are the best practices for calculating entropy for an empirically measured distribution of a continuous random variable?" That's the question I'll attempt to answer.

The good news is that the entropy of a continuous random variable is well-defined: one simply calculates an integral rather than a discrete sum:

$$ H = -\sum_{i} p_{i} \log p_{i} \Rightarrow - \int_{X} p(x) \log p(x) $$

So now your question becomes, essentially, "given a set of emprical observations of a continuous variable $x$, how can I approximate or represent the distribution $p(x)$, so that I may calculate the necessary integral?"

I can think of three different ways:

  • If you believe you have a good theoretical or mathematical model that describes the underlying shape of the distribution (for example, a Gaussian mixture model), then you can use standard parameter estimation techniques such as regression analysis or maximum likelihood estimation to come up with estimated values for the model parameters (e.g., such as Gaussian mean, variance, and mixture component weights, or whatever other parameters control the shape of the underlying model) that are the most consistent with the data that you actually observed. Once you know those parameters, that information effectively defines $p(x)$, and you can simply perform the integral by whatever means you prefer (e.g., an analytic calculation, or a numerical integral, or whatever).
  • You can generate a kernel density estimate, as you are already doing, and simply perform a numerical integration across all non-zero output values (i.e., across the so-called support for the distribution).
  • You can calculate a histogram of the observed data, and then calculate the entropy by summing over the bins of the histogram. (Technically you could argue that a histogram is really just a trivial example of a kernel density estimate anyway--one in which the kernel function is a simple boxcar, and the kernel bandwidth is therefore identical to the histogram bin width.)

Of these three methods, I find the first the most aesthetically appealing, provided one actually has a good theoretical model to explain the underlying data. However, coming up with a good theoretical model and then fitting it to the observed data isn't always possible, and in those cases, the other two methods are also perfectly defensible, as long as they are implemented in a way that generates a "reasonable" estimate of the underlying probability distribution $p(x)$. To generate a reasonable approximation of the distribution, the most important thing to do is simply make sure that you choose a reasonable value for the bandwidth (e.g., using this rule of thumb) if you're using a kernel density estimation technique, or the bin width (using one of these rules of thumb) if you're representing $p(x)$ as a histogram.

In summary, yes there are other techniques besides kernel density estimation that you can use to derive an estimate for $p(x)$ and calculate the entropy, but the way that you are already doing it is also perfectly valid, provided you're using a typical kernel function and setting the bandwidth value appropriately.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.