1
$\begingroup$

I'm considering how to measure mutual information between layers in deep neural network.

For example, in a MNIST dataset, with few layers of network. I simply flatten each layer to 1d array and calculate their entropy. And the mutual information should be the different between two layers' entropy.

This is for single input. And I just randomly choose, say, 1000 inputs from the whole dataset, and calculate the avg mutual information of those inputs.

It's the simplest way I can come out with. Is it reasonable? Or any better advice?

$\endgroup$
1
  • $\begingroup$ Would this be the mutual information between two multivariate distributions? $\endgroup$
    – Dave
    Commented Nov 9, 2022 at 13:26

2 Answers 2

1
$\begingroup$

maybe the following papers are helpful for you. You can find these papers in Google Scholar. And I think that the method in [3] is the most popular so far.

[1] Gabrié, Marylou, et al. "Entropy and mutual information in models of deep neural networks." Advances in Neural Information Processing Systems 31 (2018).

[2] Hild, Kenneth E., et al. "Feature extraction using information-theoretic learning." IEEE Transactions on Pattern Analysis and Machine Intelligence 28.9 (2006): 1385-1392.

[3] Belghazi, Mohamed Ishmael, et al. "Mutual information neural estimation." International conference on machine learning. PMLR, 2018.

$\endgroup$
2
  • $\begingroup$ Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. $\endgroup$
    – utobi
    Commented Nov 9, 2022 at 12:49
  • $\begingroup$ While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. - From Review $\endgroup$
    – mkt
    Commented Nov 9, 2022 at 13:22
0
$\begingroup$

I don't have enough reputation for a comment, hence I add it as an answer. ai zhonguo's answer is correct. But one also has to consider that usually, neural networks layers are deterministic transformations of their input, hence you will have an infinitely large mutual information (see e.g. Mutual information between $X$ and $f(X)$). Using Bayesian neural networks or some other ways to include randomness could alleviate the issue. Another idea would be to use an alternative (but related) information measure such as $\mathcal{V}$-information [1].

I once did an internship, where I tried something similar and found MINE [2] (from ai zhonguo's answer) also has serious shortcomings, you can find my technical report here if you are interested.

[1] Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon. A theory of usable information under computational constraints, ICLR 2020

[2] Belghazi, Mohamed Ishmael, et al. "Mutual information neural estimation." International conference on machine learning. PMLR, 2018

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.