In 2015, Tishby and Zaslavsky published a well-known paper claiming that the so-called Information Bottleneck Principle could be used to understand some behaviour of deep neural networks. In a more recent (April 2017) paper, Schwartz-Ziv and Tishby expand on these claims, in particular visualising some of the results.
Later in 2017, a critical paper by Saxe et al. was posted on the OpenReview website (with revisions as recent as 2 weeks ago). It claims that many of the claims made in the Schwartz-Ziv-Tishby paper don't hold up, or at least not in the generality claimed. In fact, if I am reading them right, they claim that the visualised result are an artifact of the choice of activation function -- something that should not matter according to the theory.
However, in the comments, Schwartz-Ziv and Tishby show up with a long list of commentary on the critical paper, saying that the criticism misses the mark. To this in turn the authors of the critical paper respond, but perhaps the conversation is not yet finished.
I am interested in starting a research project into the deep learning aspects of the information bottleneck, but worried that I am going to waste time learning something that has already been 'refuted'. Therefore, my question is:
What is the current expert opinion on the applicability of the Information Bottleneck Principle to understanding Deep Learning?
In particular, I am interested in research on the subject other than what I have linked, and commentary by experts (either directly or indirectly).