Entropy of a matrix with Bernoulli distributed (binary entries) row-vectors

Question

The entropy $H[x]$ of a Bernoulli distributed binary random variable $x$ is given by : $$ H[x]=−θlnθ−(1−θ)ln(1−θ) $$

where $$ p(x=1∣θ)= \theta \\ p(x=0∣θ)=1−θ $$

Now, suppose I have a vector as so:

$$ \mathbf{x} = [1,0,1,1,0] $$

where all the elements are sampled according to the Bernoulli distribution.

Consider further now that I have a matrix, with these types of rows, where all the rows are exchangeable:

$$ \mathbf{X} \triangleq [\mathbf{x}_1,\ldots,\mathbf{x}_4]^{\mathsf{T}} = \begin{bmatrix} 1 & 0 & 0 & 1 & 1 \\ 0 & 0 & 0 & 1 & 1 \\ 1 & 1 & 0 & 1 & 1 \\ 0 & 1 & 0 & 1 & 0 \end{bmatrix} $$

Hence, given that the rows are independent from one another, and that their order does not change the overall, probability of this matrix (as an assumption); how do we calculate the entropy of this matrix?

EDIT: in general I am not considering square matrices.

Answer 1

Let the entropy of a matrix be the sum of the entropies of the eigenvalues. So if that's the case then for the matrix $\mathbf{X}$. Formally, we need to calculate the following: \begin{equation} H(\mathbf{X})=-\sum_i \lambda_i\ln\lambda_i \end{equation}

From $SVD$ we get, $\lambda_1=2.95,\lambda_2=1.21,\lambda_3=0.83$ and $\lambda_4=0.34$. To get the answer, plugin the values of $\lambda_i$ for $i=1,2,3,4$ in the above formula.

Response to the comment below

You may shuffle the rows, but this will not change the eigenvalues.

From a theoretical point of view if a random variable is generated from independent identically distributed sequence then it is also exchangeable. In other words, independence implies exchangeability, but the converse is not true.

Empirically,

a<-c(1,0,1,0)
b<-c(0,0,1,1)
c<-c(1,1,1,1)
d<-c(1,1,1,0)

X<-cbind(a,b,c,d)
X <- X[sample(nrow(X)),]
svd(X)

we get the same eigenvalues as above $\lambda_1=2.95,\lambda_2=1.21,\lambda_3=0.83$ and $\lambda_4=0.34$.

Response to the edit

Considering a rectangular matrix $R$.

R<- as.matrix(data.frame(c(4,7,-1,8), c(-5,-2,4,2), c(-1,3,-3,6)))
R <- R[sample(nrow(R)),]
svd(R)

So, $\lambda_1=13.16$, $\lambda_2=6.99$ and $\lambda_3=3.43$. To get the entropy plugin the values of $\lambda$ in the above-given definition of entropy. You may shuffle the rectangular matrix, but if the data is i.i.d than eigenvalues will be the same.