3
$\begingroup$

I am looking for a measure of entropy over multiple random variables, each with values between 0 and 1.

Intuitively, it seems possible to talk about the expected value of information of several variables, which is entropy, but I am not sure how to go about.

I know that in the continuous case, for a single value, we should look at the differential entropy, given by $$ - \int_0^1 f(x) log(f(x)) \partial x $$ , where $ f(x) $ is the probability distribution function, which in my case is a Beta Distribution, fitted to the data.

I am fitting a Dirichlet distribution for the multiple variable case, since it is the multivariate generalization of the beta distribution.

How do we measure the entropy of such a joint probability density function?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
3
$\begingroup$

Even in the univariate case there is a problem with this definition. If we define the entropy of the distribution of the random variable $X$ to be $$ - \int f_X(x) \log f_X(x) \, dx \qquad (*) $$ and transform using a one-to-one smooth $g$, doing $Y=g(X)$, then $$ - \int f_Y(y) \log f_Y(y) \, dy $$ won't, in general, be equal to $(*)$ (check it out doing the transformation: that Jacobian in the argument of the $\log$ shouldn't be there). Hence, the real number $(*)$ can't be a measure of the entropy of the distribution of $X$, since $X$ and $Y$ contain the same information. One thing people do is to introduce a reference density $r$, and use the definition $$ - \int f_X(x) \log \left( \frac{f_X(x)}{r(x)} \right) dx \, . $$ It is easy to check that this transforms properly. Finally, it doesn't matter in these definitions if $X$ is a random variable, or a random vector $(X_1,\dots,X_k)$. In the multivariate case you can define $$ - \int f_{X_1,\dots,X_k}(x_1,\dots,x_k) \log \left( \frac{f_{X_1,\dots,X_k}(x_1,\dots,x_k)}{r(x_1,\dots,x_k)} \right) dx_1\dots\,dx_k \, . $$

Improve this answer
$\endgroup$
5
  • $\begingroup$ Actually my goal is to maximize the entropy of the output of an artificial neural network, by applying a multiplication to the weights, in order for the outputs to be more uniformly distributed. This weight-multiplication actually constitutes a one-to-one smooth transformation. My thoughts were that if the outputs are always nearly 0 or 1, the information content would be lower than when the outputs are uniformly distributed between 0 and 1. I don't share your intuition that $X$ and $Y$ contain the same information. $\endgroup$
    – Angelorf
    Commented Feb 11, 2014 at 16:46
  • $\begingroup$ OK, no problem. But it's not my intuition, it's a mathematical fact: the sigma-fields generated by $X$ and $Y$ are exactly the same. $\endgroup$
    – Zen
    Commented Feb 11, 2014 at 22:10
  • $\begingroup$ I'm sorry for suggesting it. If I'm right the sigma-fields of $X$ and $Y$ are both all measurable subsets of the interval $[0,1]$. I think the given standard definition of entropy is what I need. But your point is that the formula can't rightfully be called 'entropy', since it cannot be the expected value of information; let's say it's a formula for the information density (a term which I have just thought up). Is there an official term for what the formula actually computes? $\endgroup$
    – Angelorf
    Commented Feb 12, 2014 at 9:33
  • $\begingroup$ Hi, Angelorf. The last formula in my answer is (minus) the Kullback-Leibler divergence between $f$ and the reference density $r$. The point is that entropy is a relative concept, but in the finite case we always have good old faithful uniform as a reference. In general (like when the support of $f$ is $\mathbb{R}^k$), we have to choose $r$. $\endgroup$
    – Zen
    Commented Feb 12, 2014 at 10:42
  • $\begingroup$ In your case, it seems OK to take $r$ as uniform on the simplex (aka $\mathrm{Dirichlet}(1,1,\dots,1)$). Note that if you have simulated values $X^{(1)},\dots,X^{(N)}$ from $f$, then $(1/N)\sum_{i=1} f_X(X^{(i)})/r(X^{(i)})$ converges a.s to what you need, because the last expression in my answer is $\mathrm{E}[f_X(Y)/r(Y)]$, in which $Y\sim f$. $\endgroup$
    – Zen
    Commented Feb 12, 2014 at 10:53
1
$\begingroup$

I do not know much about this but for the multivariate case, should it not simply be

$$ H(x, y) = -\int p(x, y) \log p(x, y) dx dy $$

Improve this answer
$\endgroup$
2
  • $\begingroup$ That's my guess as well. Don't know for sure though. $\endgroup$
    – Angelorf
    Commented Feb 11, 2014 at 12:21
  • $\begingroup$ I think it should be. It follows from the definition of entropy. $\endgroup$
    – Luca
    Commented Feb 11, 2014 at 12:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.