0
$\begingroup$

I would like to estimate MI between two variables X and Y of shape (nXd) and (nX1) respectively. As X is multi-dimensional (dimension = d) and Y is of dimsnension=1, is it a correct approach of the finding MI between each dimension of X against Y and adding each dimension's MI to get final MI? Sample example:

X = rand[N,d]
Y = rand[N,]

for i=1 to d:
  mi_total += calculate_MI( X[:,i] , Y) #X[:,i] i_th dimension of X

# X[:,i].shape = (N,1) = Y.shape

Any help and correction is very much appreciated and Thanks in advance :)

$\endgroup$
0
$\begingroup$

Theoretically speaking, the following quantities are not equal in general, i.e. $$I(\mathbf{X};Y)\neq \sum_{i=1}^dI(X_i;Y)$$ Because, we are dropping out any dependence relationship. For example, if $\mathbf{X}=[X_1,X_1]$ (i.e. $X_2=X_1$), then the mutual information between $\mathbf{X}$ and $Y$ would be $I(X_1;Y)$, not $I(X_1;Y)+I(X_2;Y)=2I(X_1;Y)$. However, if the random variables inside $\mathbf{X}$ are independent, you can sum their individual MIs, as you've done in your code. I couldn't capture yours, but in many coding languages rand is assumed to create independent samples, in which you can really sum MIs.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you. X in my case is a dataset with d dimension, so I reckon it is safe assuming that each dimension is independent of each other and also the reason for using rand. I think my doubt is cleared :) $\endgroup$ – Vigneswaran C Jun 18 '19 at 16:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.