I am trying to implement Information Bottleneck principle. In which one of the observation is Mutual Information between Input data X and Hidden layer's out H keeps reducing as we go deeper. In other words, Information about X (synergistic) is high in shallow layer and become unique & redundant in deeper layers. Hence I(X,H) from the first layer to the last layer looks like,

MI(X,H) trend Source: mentioned in this post

Now, when I try to implement, I am having trouble in estimating MI with the basic problem of dimensionality inequality.

   # when a sample in X is given as input to the first layer
   #out.shape = [ no_of_neurons_in_first_layer x 1]
   # to get amount of X's Information mi[j] provided by a single neuron(jth) 
   loop all X: i -> 1 to N:
      out = activation(X[i]) 
      mi[j] += MI(out[j] , X[i]) #estimate MI of out[j](single Random Variable
                                 #and X[i] (a sequence of RVs)

How can I calculate MI between sequence of RVs and single RV? Is it correct to quantize them and do the histogram method (in spite only one member from one group)? Any insight and help would be appreciated. Thanks in advance :)


The information bottleneck conclusions are known to be sensitive to implementation choices, cf. for example https://openreview.net/forum?id=ry_WPG-A-.

If you want a starting point, you can look at the official github for a later paper from that group.

This gist had some simple and as far as I can tell correct and efficient code for estimating mutual information.

| cite | improve this answer | |
  • 1
    $\begingroup$ It would be helpful if you could summarize the main content of the materials that you link to. The purpose of this website is to build a durable repository of information. As it stands, if the links go dead, your answer will have relatively little information. $\endgroup$ – Sycorax Jun 15 '19 at 15:37
  • $\begingroup$ This is a matter of some controvery, I don't want to hamhandedly attempt a summary. My pointers are durable-by-construction. $\endgroup$ – user39430 Jun 15 '19 at 15:40
  • $\begingroup$ Yes, I agree with the fact that the principle is sensitive to implementation. However, that is mainly involved with other factors like selection of activation function, an unclear connection between compression and generalization phase etc. The estimation method for Mutual Information remains robust and gives the same result irrespective of the method employed (Even mentioned in one of the replies by the Authors), So it would be great if you help to provide insight to estimate MI using simple binning procedure, which is the exact thing the question demands. Thanks:) $\endgroup$ – Vigneswaran C Jun 17 '19 at 6:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.