I posted this question in the stackexchange mathematics forum without any reponse. Maybe it was the wrong forum, so I try it here.

I tried to compute the multi-information (MI) $I[\mathbf U] = \sum_{i=1}^n H[U_i] - H[\mathbf U]$ of a uniformly distributed RV on the $L_1$ unit sphere. Below are my calculations. What confuses me is, that for $n=2$ the MI should be large since I can narrow down the value of $U_1$ to two possible values if I have $U_2$. However, my formula gives me zero. For larger values of $n$ it even becomes negative (which should not happen since the MI is always positive). What am I doing wrong?

Calculations A uniformly distributed random variable on the $L_{1}$ unit sphere has the stochastic representation $\boldsymbol{U}\stackrel{d}{=}\boldsymbol{Z}\odot\boldsymbol{X}$ ($\odot$ is the elementwise multiplication) where $Z_{i}\in\{-1,1\}$, $Z_{i}$ Bernoulli with $p=\frac{1}{2}$, and $\boldsymbol{X}$ has a Dirichlet distribution with parameters $\alpha_{1}=...=\alpha_{n}=1$, and $\boldsymbol{Z}$ and $\boldsymbol{X}$ are independent.

To compute the multi-information of $\boldsymbol{U}$, I use the independence of $\boldsymbol{Z}$ and $\boldsymbol{X}$ to get \begin{eqnarray*} I\left[\boldsymbol{U}\right] & = & I\left[\boldsymbol{Z}\right]+I\left[\boldsymbol{X}\right] = I\left[\boldsymbol{X}\right] \end{eqnarray*} since the coefficients of $\boldsymbol{Z}$ are independent.

The marginal distribution of a the Dirichlet distribution is a $\beta$-distribution with parameters $\alpha=1$ and $\beta=n-1$. Therefore, the marginal entropy is \begin{eqnarray*} H[X_{i}] & = & \log B(\alpha,\beta)-(\alpha-1)\psi\left(\alpha\right)-(\beta-1)\psi\left(\beta\right)+(\alpha+\beta-2)\psi(\alpha+\beta)\\ & = & \log\frac{\Gamma\left(1\right)\Gamma\left(n-1\right)}{\Gamma\left(n\right)}-(n-2)\psi\left(n-1\right)+(n-2)\psi(n). \end{eqnarray*}

The entropy of this particular Dirichlet distribution is \begin{eqnarray*} H\left[\boldsymbol{X}\right] & = & \log\frac{\Gamma\left(1\right)^{n}}{\Gamma\left(n\right)}=-\log\Gamma\left(n\right). \end{eqnarray*} Therefore, the multi-information is given by \begin{eqnarray*} I\left[\boldsymbol{X}\right] & = & n\log\Gamma\left(n-1\right)-n\log\Gamma\left(n\right)-n(n-2)\psi\left(n-1\right)+n(n-2)\psi(n)+\log\Gamma\left(n\right). \end{eqnarray*}


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.