The entropy $H[x]$ of a Bernoulli distributed binary random variable $x$ is given by : $$ H[x]=−θlnθ−(1−θ)ln(1−θ) $$

where $$ p(x=1∣θ)= \theta \\ p(x=0∣θ)=1−θ $$

Now, suppose I have a vector as so:

$$ \mathbf{x} = [1,0,1,1,0] $$

where all the elements are sampled according to the Bernoulli distribution.

Consider further now that I have a matrix, with these types of rows, where all the rows are exchangeable:

$$ \mathbf{X} \triangleq [\mathbf{x}_1,\ldots,\mathbf{x}_4]^{\mathsf{T}} = \begin{bmatrix} 1 & 0 & 0 & 1 & 1 \\ 0 & 0 & 0 & 1 & 1 \\ 1 & 1 & 0 & 1 & 1 \\ 0 & 1 & 0 & 1 & 0 \end{bmatrix} $$

Hence, given that the rows are independent from one another, and that their order does not change the overall, probability of this matrix (as an assumption); how do we calculate the entropy of this matrix?

EDIT: in general I am not considering square matrices.


Let the entropy of a matrix be the sum of the entropies of the eigenvalues. So if that's the case then for the matrix $\mathbf{X}$. Formally, we need to calculate the following: \begin{equation} H(\mathbf{X})=-\sum_i \lambda_i\ln\lambda_i \end{equation}

From $SVD$ we get, $\lambda_1=2.95,\lambda_2=1.21,\lambda_3=0.83$ and $\lambda_4=0.34$. To get the answer, plugin the values of $\lambda_i$ for $i=1,2,3,4$ in the above formula.

Response to the comment below

You may shuffle the rows, but this will not change the eigenvalues.

From a theoretical point of view if a random variable is generated from independent identically distributed sequence then it is also exchangeable. In other words, independence implies exchangeability, but the converse is not true.



X <- X[sample(nrow(X)),]

we get the same eigenvalues as above $\lambda_1=2.95,\lambda_2=1.21,\lambda_3=0.83$ and $\lambda_4=0.34$.

Response to the edit

Considering a rectangular matrix $R$.

R<- as.matrix(data.frame(c(4,7,-1,8), c(-5,-2,4,2), c(-1,3,-3,6)))
R <- R[sample(nrow(R)),]

So, $\lambda_1=13.16$, $\lambda_2=6.99$ and $\lambda_3=3.43$. To get the entropy plugin the values of $\lambda$ in the above-given definition of entropy. You may shuffle the rectangular matrix, but if the data is i.i.d than eigenvalues will be the same.

  • $\begingroup$ That's very interesting! But if I shuffle the rows around, then I'll get different eigenvalues presumably, and so that measure is not exchangeable since it is not invariant to row permutations - I think? $\endgroup$ – Astrid Apr 25 '18 at 21:59
  • 1
    $\begingroup$ Please see the updated answer. $\endgroup$ – Waqas Apr 26 '18 at 7:14
  • $\begingroup$ Ahhh. I see, nice! $\endgroup$ – Astrid Apr 26 '18 at 8:18
  • $\begingroup$ So, does this answer your question? $\endgroup$ – Waqas Apr 26 '18 at 8:49
  • $\begingroup$ Not yet I am afraid, having read about it more, it is not quite what I am looking for. Stupidly, I did not say that I was considering non-square matrices. I have updated the question for that now (though I am trying to figure out how I can use square matrices with Von Neumann entropy now). $\endgroup$ – Astrid Apr 26 '18 at 10:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.