# Entropy of a matrix with Bernoulli distributed (binary entries) row-vectors

The entropy $H[x]$ of a Bernoulli distributed binary random variable $x$ is given by : $$H[x]=−θlnθ−(1−θ)ln(1−θ)$$

where $$p(x=1∣θ)= \theta \\ p(x=0∣θ)=1−θ$$

Now, suppose I have a vector as so:

$$\mathbf{x} = [1,0,1,1,0]$$

where all the elements are sampled according to the Bernoulli distribution.

Consider further now that I have a matrix, with these types of rows, where all the rows are exchangeable:

$$\mathbf{X} \triangleq [\mathbf{x}_1,\ldots,\mathbf{x}_4]^{\mathsf{T}} = \begin{bmatrix} 1 & 0 & 0 & 1 & 1 \\ 0 & 0 & 0 & 1 & 1 \\ 1 & 1 & 0 & 1 & 1 \\ 0 & 1 & 0 & 1 & 0 \end{bmatrix}$$

Hence, given that the rows are independent from one another, and that their order does not change the overall, probability of this matrix (as an assumption); how do we calculate the entropy of this matrix?

EDIT: in general I am not considering square matrices.

Let the entropy of a matrix be the sum of the entropies of the eigenvalues. So if that's the case then for the matrix $$\mathbf{X}$$. Formally, we need to calculate the following: $$$$H(\mathbf{X})=-\sum_i \lambda_i\ln\lambda_i$$$$

From $$SVD$$ we get, $$\lambda_1=2.95,\lambda_2=1.21,\lambda_3=0.83$$ and $$\lambda_4=0.34$$. To get the answer, plugin the values of $$\lambda_i$$ for $$i=1,2,3,4$$ in the above formula.

Response to the comment below

You may shuffle the rows, but this will not change the eigenvalues.

From a theoretical point of view if a random variable is generated from independent identically distributed sequence then it is also exchangeable. In other words, independence implies exchangeability, but the converse is not true.

Empirically,

a<-c(1,0,1,0)
b<-c(0,0,1,1)
c<-c(1,1,1,1)
d<-c(1,1,1,0)

X<-cbind(a,b,c,d)
X <- X[sample(nrow(X)),]
svd(X)


we get the same eigenvalues as above $$\lambda_1=2.95,\lambda_2=1.21,\lambda_3=0.83$$ and $$\lambda_4=0.34$$.

Response to the edit

Considering a rectangular matrix $$R$$.

R<- as.matrix(data.frame(c(4,7,-1,8), c(-5,-2,4,2), c(-1,3,-3,6)))
R <- R[sample(nrow(R)),]
svd(R)


So, $$\lambda_1=13.16$$, $$\lambda_2=6.99$$ and $$\lambda_3=3.43$$. To get the entropy plugin the values of $$\lambda$$ in the above-given definition of entropy. You may shuffle the rectangular matrix, but if the data is i.i.d than eigenvalues will be the same.

• That's very interesting! But if I shuffle the rows around, then I'll get different eigenvalues presumably, and so that measure is not exchangeable since it is not invariant to row permutations - I think? – Astrid Apr 25 '18 at 21:59
• Please see the updated answer. – Waqas Apr 26 '18 at 7:14
• Ahhh. I see, nice! – Astrid Apr 26 '18 at 8:18
• So, does this answer your question? – Waqas Apr 26 '18 at 8:49
• Not yet I am afraid, having read about it more, it is not quite what I am looking for. Stupidly, I did not say that I was considering non-square matrices. I have updated the question for that now (though I am trying to figure out how I can use square matrices with Von Neumann entropy now). – Astrid Apr 26 '18 at 10:01