# Multidimensional Differential Entropy

I am looking for a measure of entropy over multiple random variables, each with values between 0 and 1.

Intuitively, it seems possible to talk about the expected value of information of several variables, which is entropy, but I am not sure how to go about.

I know that in the continuous case, for a single value, we should look at the differential entropy, given by $$- \int_0^1 f(x) log(f(x)) \partial x$$ , where $f(x)$ is the probability distribution function, which in my case is a Beta Distribution, fitted to the data.

I am fitting a Dirichlet distribution for the multiple variable case, since it is the multivariate generalization of the beta distribution.

How do we measure the entropy of such a joint probability density function?

Even in the univariate case there is a problem with this definition. If we define the entropy of the distribution of the random variable $X$ to be $$- \int f_X(x) \log f_X(x) \, dx \qquad (*)$$ and transform using a one-to-one smooth $g$, doing $Y=g(X)$, then $$- \int f_Y(y) \log f_Y(y) \, dy$$ won't, in general, be equal to $(*)$ (check it out doing the transformation: that Jacobian in the argument of the $\log$ shouldn't be there). Hence, the real number $(*)$ can't be a measure of the entropy of the distribution of $X$, since $X$ and $Y$ contain the same information. One thing people do is to introduce a reference density $r$, and use the definition $$- \int f_X(x) \log \left( \frac{f_X(x)}{r(x)} \right) dx \, .$$ It is easy to check that this transforms properly. Finally, it doesn't matter in these definitions if $X$ is a random variable, or a random vector $(X_1,\dots,X_k)$. In the multivariate case you can define $$- \int f_{X_1,\dots,X_k}(x_1,\dots,x_k) \log \left( \frac{f_{X_1,\dots,X_k}(x_1,\dots,x_k)}{r(x_1,\dots,x_k)} \right) dx_1\dots\,dx_k \, .$$

• Actually my goal is to maximize the entropy of the output of an artificial neural network, by applying a multiplication to the weights, in order for the outputs to be more uniformly distributed. This weight-multiplication actually constitutes a one-to-one smooth transformation. My thoughts were that if the outputs are always nearly 0 or 1, the information content would be lower than when the outputs are uniformly distributed between 0 and 1. I don't share your intuition that $X$ and $Y$ contain the same information. – Angelorf Feb 11 '14 at 16:46
• OK, no problem. But it's not my intuition, it's a mathematical fact: the sigma-fields generated by $X$ and $Y$ are exactly the same. – Zen Feb 11 '14 at 22:10
• I'm sorry for suggesting it. If I'm right the sigma-fields of $X$ and $Y$ are both all measurable subsets of the interval $[0,1]$. I think the given standard definition of entropy is what I need. But your point is that the formula can't rightfully be called 'entropy', since it cannot be the expected value of information; let's say it's a formula for the information density (a term which I have just thought up). Is there an official term for what the formula actually computes? – Angelorf Feb 12 '14 at 9:33
• Hi, Angelorf. The last formula in my answer is (minus) the Kullback-Leibler divergence between $f$ and the reference density $r$. The point is that entropy is a relative concept, but in the finite case we always have good old faithful uniform as a reference. In general (like when the support of $f$ is $\mathbb{R}^k$), we have to choose $r$. – Zen Feb 12 '14 at 10:42
• In your case, it seems OK to take $r$ as uniform on the simplex (aka $\mathrm{Dirichlet}(1,1,\dots,1)$). Note that if you have simulated values $X^{(1)},\dots,X^{(N)}$ from $f$, then $(1/N)\sum_{i=1} f_X(X^{(i)})/r(X^{(i)})$ converges a.s to what you need, because the last expression in my answer is $\mathrm{E}[f_X(Y)/r(Y)]$, in which $Y\sim f$. – Zen Feb 12 '14 at 10:53

I do not know much about this but for the multivariate case, should it not simply be

$$H(x, y) = -\int p(x, y) \log p(x, y) dx dy$$

• That's my guess as well. Don't know for sure though. – Angelorf Feb 11 '14 at 12:21
• I think it should be. It follows from the definition of entropy. – Luca Feb 11 '14 at 12:24