2
$\begingroup$

The entropy $H[x]$ of a Bernoulli distributed binary random variable $x$ is given by : $$ H[x]=−θlnθ−(1−θ)ln(1−θ) $$

where $$ p(x=1∣θ)= \theta \\ p(x=0∣θ)=1−θ $$

Now, suppose I have a vector as so:

$$ \mathbf{x} = [1,0,1,1,0] $$

where all the elements are sampled according to the Bernoulli distribution.

Consider further now that I have a matrix, with these types of rows, where all the rows are exchangeable:

$$ \mathbf{X} \triangleq [\mathbf{x}_1,\ldots,\mathbf{x}_4]^{\mathsf{T}} = \begin{bmatrix} 1 & 0 & 0 & 1 & 1 \\ 0 & 0 & 0 & 1 & 1 \\ 1 & 1 & 0 & 1 & 1 \\ 0 & 1 & 0 & 1 & 0 \end{bmatrix} $$

Hence, given that the rows are independent from one another, and that their order does not change the overall, probability of this matrix (as an assumption); how do we calculate the entropy of this matrix?

EDIT: in general I am not considering square matrices.

$\endgroup$

1 Answer 1

2
$\begingroup$

Let the entropy of a matrix be the sum of the entropies of the eigenvalues. So if that's the case then for the matrix $\mathbf{X}$. Formally, we need to calculate the following: \begin{equation} H(\mathbf{X})=-\sum_i \lambda_i\ln\lambda_i \end{equation}

From $SVD$ we get, $\lambda_1=2.95,\lambda_2=1.21,\lambda_3=0.83$ and $\lambda_4=0.34$. To get the answer, plugin the values of $\lambda_i$ for $i=1,2,3,4$ in the above formula.

Response to the comment below

You may shuffle the rows, but this will not change the eigenvalues.

From a theoretical point of view if a random variable is generated from independent identically distributed sequence then it is also exchangeable. In other words, independence implies exchangeability, but the converse is not true.

Empirically,

a<-c(1,0,1,0)
b<-c(0,0,1,1)
c<-c(1,1,1,1)
d<-c(1,1,1,0)

X<-cbind(a,b,c,d)
X <- X[sample(nrow(X)),]
svd(X)

we get the same eigenvalues as above $\lambda_1=2.95,\lambda_2=1.21,\lambda_3=0.83$ and $\lambda_4=0.34$.

Response to the edit

Considering a rectangular matrix $R$.

R<- as.matrix(data.frame(c(4,7,-1,8), c(-5,-2,4,2), c(-1,3,-3,6)))
R <- R[sample(nrow(R)),]
svd(R)

So, $\lambda_1=13.16$, $\lambda_2=6.99$ and $\lambda_3=3.43$. To get the entropy plugin the values of $\lambda$ in the above-given definition of entropy. You may shuffle the rectangular matrix, but if the data is i.i.d than eigenvalues will be the same.

$\endgroup$
9
  • $\begingroup$ That's very interesting! But if I shuffle the rows around, then I'll get different eigenvalues presumably, and so that measure is not exchangeable since it is not invariant to row permutations - I think? $\endgroup$
    – Astrid
    Apr 25, 2018 at 21:59
  • 1
    $\begingroup$ Please see the updated answer. $\endgroup$
    – Waqas
    Apr 26, 2018 at 7:14
  • $\begingroup$ Ahhh. I see, nice! $\endgroup$
    – Astrid
    Apr 26, 2018 at 8:18
  • $\begingroup$ So, does this answer your question? $\endgroup$
    – Waqas
    Apr 26, 2018 at 8:49
  • $\begingroup$ Not yet I am afraid, having read about it more, it is not quite what I am looking for. Stupidly, I did not say that I was considering non-square matrices. I have updated the question for that now (though I am trying to figure out how I can use square matrices with Von Neumann entropy now). $\endgroup$
    – Astrid
    Apr 26, 2018 at 10:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.