29
$\begingroup$

Han et al. (2015) used a method of iterative pruning to reduce their network to only 10% of its original size with no loss of accuracy by removing weights with very low values, since these changed very little. As someone new to the machine learning area, why wouldn't you do this (unless your network is already very small)? It seems to me that for deep learning your network would be smaller, faster, more energy efficient, etc. at no real cost. Should we all use this method for larger neural networks?

$\endgroup$
40
$\begingroup$

Pruning is indeed remarkably effective and I think it is pretty commonly used on networks which are "deployed" for use after training.

The catch about pruning is that you can only increase efficiency, speed, etc. after training is done. You still have to train with the full size network. Most computation time throughout the lifetime of a model's development and deployment is spent during development: training networks, playing with model architectures, tweaking parameters, etc. You might train a network several hundred times before you settle on the final model. Reducing computation of the deployed network is a drop in the bucket compared to this.

Among ML researchers, we're mainly trying to improve training techniques for DNN's. We usually aren't concerned with deployment, so pruning isn't used there.

There is some research on utilizing pruning techniques to speed up network training, but not much progress has been made. See, for example, my own paper from 2018 which experimented with training on pruned and other structurally sparse NN architectures: https://arxiv.org/abs/1810.00299

$\endgroup$
1
27
$\begingroup$

In addition to the points raised in the other answers, a pruned network may not be faster. Common machine learning frameworks have very efficient optimizations for computing dense matrix multiplications (i.e. normal, unpruned layers), but those algorithms can't take any additional advantage of the fact that some weights are set to 0 (because they are pruned).

So the result of pruning is often a neural network that is smaller, but no faster and has worse performance. In many cases, better performance is more important than a smaller model size, so pruning is not useful in those cases.

Note that pruned networks could be faster if 1. an overwhelmingly large fraction of weights were pruned away, in which case sparse matrix multiplication algorithms might start being faster; or 2. (in CNNs; I'm not sure off the top of my head if this is applicable to other architectures) if pruning was not weight-level but rather channel-level (so either an entire channel is pruned all at once or the whole channel is left as is), which does work with the optimizations; or 3. given specialized hardware or ML frameworks.

$\endgroup$
1
  • $\begingroup$ "So the result of pruning is often a neural network that is smaller, but no faster and has worse performance" - I've heard some claims that pruning a network and retraining it might be an effective way to remove biases and overfitting. Do you happen to know what's happened to those claims? $\endgroup$ Jun 29 '20 at 21:28
10
$\begingroup$

As mentioned previously, you need to train on large networks in order to prune them. There are some theories as to why, but the one I'm most familiar with is the "golden ticket" theory. Presented in "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks" by Jonathan Frankle, Michael Carbin the golden ticket theory of neural networks asserts that there is a subset of the network which is already very close and what training does is to find and slightly improve this subset of the network, while downplaying the wrong parts of the network. A real-life analogy of this is that only a few of your lottery tickets will be worth anything but you need to buy a lot in order to find them.

There is a connection to the original rationale behind dropout: Train many networks 'in parallel' and some of the time you will be training the only the golden ticket network.

$\endgroup$
3
  • 1
    $\begingroup$ This is a great reference paper! You mention that the winning ticket weights are "already very close", but this contradicts the paper and I think it is important to clarify. In Appendix F5, the authors actually state that "winning ticket weights tend to change by a larger amount then weights in the rest of the network, evidence that does not support the rationale that winning tickets are already close to the optimum" $\endgroup$ Jun 9 at 20:38
  • $\begingroup$ Good catch, I believe I was actually thinking of the paper "Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask" (arxiv.org/abs/1905.01067) which finds that you don't need to train them you only need to mask them. $\endgroup$ Jun 9 at 22:43
  • 1
    $\begingroup$ Ah interesting, I'll have to check that one out. Been meaning to read some of the related literature that came out of the original paper (arxiv.org/pdf/1803.03635.pdf) $\endgroup$ Jun 10 at 14:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.