I am trying to implement Information Bottleneck principle. In which one of the observation is Mutual Information between Input data X and Hidden layer's out H keeps reducing as we go deeper. In other words, Information about X (synergistic) is high in shallow layer and become unique & redundant in deeper layers. Hence I(X,H) from the first layer to the last layer looks like,
Source: mentioned in this post
Now, when I try to implement, I am having trouble in estimating MI with the basic problem of dimensionality inequality.
# when a sample in X is given as input to the first layer #out.shape = [ no_of_neurons_in_first_layer x 1] # to get amount of X's Information mi[j] provided by a single neuron(jth) loop all X: i -> 1 to N: out = activation(X[i]) mi[j] += MI(out[j] , X[i]) #estimate MI of out[j](single Random Variable #and X[i] (a sequence of RVs)
How can I calculate MI between sequence of RVs and single RV? Is it correct to quantize them and do the histogram method (in spite only one member from one group)? Any insight and help would be appreciated. Thanks in advance :)