Why low rank expansions can exploit the redundancy that exist between different feature channels and filters?

Question

I read Jaderberg et al., 2014 paper about Speeding up Convolutional Neural Network with Low Rank Expansions. In the introduction, it is written in bold font:

Our key insight is to exploit the redundancy that exists between feature channels and filters.

What is the connection? Redundancy in feature channels and filters are maybe can be imagined that there are many blank feature after training. And in this paper, they approximated the CNN filter from 4 dimension to only 3 dimension (scheme 2). I quite understand how it works after reading Speeding-up Convolutional Neural Networks using Fine-Tuned CP-Decomposition (Lebedev et al. 2915). Inside that paper there is picture about this speeding up using low rank expansion, so I kind of understand the final computation even though until now I still don't understand how to compute/approximate the lower rank filter. (this is not my question, but if someone can explain it to me, I would be very grateful)

How that linear summation of lower rank weight can exploit the redundancy? What is the logic/intuition behind this?

Answer 1

Have a look at Channel Pruning, ICCV 2017, an elegant algorithm that effectively prune channels each layer. It also helps understanding the redundancy you've mentioned. Also have a try on its github code.