how to measure mutual information in deep neural network I'm considering how to measure mutual information between layers in deep neural network.
For example, in a MNIST dataset, with few layers of network. I simply flatten each layer to 1d array and calculate their entropy. And the mutual information should be the different between two layers' entropy.
This is for single input.
And I just randomly choose, say, 1000 inputs from the whole dataset, and calculate the avg mutual information of those inputs.
It's the simplest way I can come out with. Is it reasonable? Or any better advice?
 A: maybe the following papers are helpful for you. You can find these papers in Google Scholar. And I think that the method in [3] is the most popular so far.
[1] Gabrié, Marylou, et al. "Entropy and mutual information in models of deep neural networks." Advances in Neural Information Processing Systems 31 (2018).
[2] Hild, Kenneth E., et al. "Feature extraction using information-theoretic learning." IEEE Transactions on Pattern Analysis and Machine Intelligence 28.9 (2006): 1385-1392.
[3] Belghazi, Mohamed Ishmael, et al. "Mutual information neural estimation." International conference on machine learning. PMLR, 2018.
A: I don't have enough reputation for a comment, hence I add it as an answer. ai zhonguo's answer is correct. But one also has to consider that usually, neural networks layers are deterministic transformations of their input, hence you will have an infinitely large mutual information (see e.g. Mutual information between $X$ and $f(X)$). Using Bayesian neural networks or some other ways to include randomness could alleviate the issue. Another idea would be to use an alternative (but related) information measure such as $\mathcal{V}$-information [1].
I once did an internship, where I tried something similar and found MINE [2] (from ai zhonguo's answer) also has serious shortcomings, you can find my technical report here if you are interested.
[1] Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon. A theory of usable information under computational constraints, ICLR 2020
[2] Belghazi, Mohamed Ishmael, et al. "Mutual information neural estimation." International conference on machine learning. PMLR, 2018
