# Is there current consensus on the value of the Information Bottleneck Principle to understanding Deep Learning?

In 2015, Tishby and Zaslavsky published a well-known paper claiming that the so-called Information Bottleneck Principle could be used to understand some behaviour of deep neural networks. In a more recent (April 2017) paper, Schwartz-Ziv and Tishby expand on these claims, in particular visualising some of the results.

Later in 2017, a critical paper by Saxe et al. was posted on the OpenReview website (with revisions as recent as 2 weeks ago). It claims that many of the claims made in the Schwartz-Ziv-Tishby paper don't hold up, or at least not in the generality claimed. In fact, if I am reading them right, they claim that the visualised result are an artifact of the choice of activation function -- something that should not matter according to the theory.

However, in the comments, Schwartz-Ziv and Tishby show up with a long list of commentary on the critical paper, saying that the criticism misses the mark. To this in turn the authors of the critical paper respond, but perhaps the conversation is not yet finished.

I am interested in starting a research project into the deep learning aspects of the information bottleneck, but worried that I am going to waste time learning something that has already been 'refuted'. Therefore, my question is:

What is the current expert opinion on the applicability of the Information Bottleneck Principle to understanding Deep Learning?

In particular, I am interested in research on the subject other than what I have linked, and commentary by experts (either directly or indirectly).

• I think it's worth noting that this is an active area of research, and that this is a very recent paper. The pre-publication review, peer review, and post-publication responses should be seen, in total, as an ongoing conversation about the topic, rather than any particular step in the process comprising the "last word." Or, in the Hegelian view, the dialogue you've cited comprises the thesis-antithesis components of the triad, and we have yet to arrive at synthesis. – Sycorax Mar 6 '18 at 18:19
• no consensus at all! Give it a try: I, and many others in the Deep Learning community, would like to see more work on it. Sure, there's a risk of failure, but you always have that in research. I cannot comment on the risk of "wasting time", since it depends on what you have at stake: 5 years of Ph.D. $\implies$ significant personal investment, and thus higher risk. 2 years of postdoc $\implies$ less chances of making it work, but also less to lose. But I can try to collect info to let you make a more informed investment :-) – DeltaIV Mar 7 '18 at 12:08
• PS it also depends on your career goals, which are off-topic here: as a research topic, it's much more palatable for an academic career. But if you want to work in industry, there are more fruitful topics in Deep Learning right now. This is IMO and other people in the field may beg to differ. – DeltaIV Mar 7 '18 at 12:11