Han et al. (2015) used a method of iterative pruning to reduce their network to only 10% of its original size with no loss of accuracy by removing weights with very low values, since these changed very little. As someone new to the machine learning area, why wouldn't you do this (unless your network is already very small)? It seems to me that for deep learning your network would be smaller, faster, more energy efficient, etc. at no real cost. Should we all use this method for larger neural networks?
Pruning is indeed remarkably effective and I think it is pretty commonly used on networks which are "deployed" for use after training.
The catch about pruning is that you can only increase efficiency, speed, etc. after training is done. You still have to train with the full size network. Most computation time throughout the lifetime of a model's development and deployment is spent during development: training networks, playing with model architectures, tweaking parameters, etc. You might train a network several hundred times before you settle on the final model. Reducing computation of the deployed network is a drop in the bucket compared to this.
Among ML researchers, we're mainly trying to improve training techniques for DNN's. We usually aren't concerned with deployment, so pruning isn't used there.
There is some research on utilizing pruning techniques to speed up network training, but not much progress has been made. See, for example, my own paper from 2018 which experimented with training on pruned and other structurally sparse NN architectures: https://arxiv.org/abs/1810.00299
In addition to the points raised in the other answers, a pruned network may not be faster. Common machine learning frameworks have very efficient optimizations for computing dense matrix multiplications (i.e. normal, unpruned layers), but those algorithms can't take any additional advantage of the fact that some weights are set to 0 (because they are pruned).
So the result of pruning is often a neural network that is smaller, but no faster and has worse performance. In many cases, better performance is more important than a smaller model size, so pruning is not useful in those cases.
Note that pruned networks could be faster if 1. an overwhelmingly large fraction of weights were pruned away, in which case sparse matrix multiplication algorithms might start being faster; or 2. (in CNNs; I'm not sure off the top of my head if this is applicable to other architectures) if pruning was not weight-level but rather channel-level (so either an entire channel is pruned all at once or the whole channel is left as is), which does work with the optimizations; or 3. given specialized hardware or ML frameworks.
As mentioned previously, you need to train on large networks in order to prune them. There are some theories as to why, but the one I'm most familiar with is the "golden ticket" theory. Presented in "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks" by Jonathan Frankle, Michael Carbin the golden ticket theory of neural networks asserts that there is a subset of the network which is already very close and what training does is to find and slightly improve this subset of the network, while downplaying the wrong parts of the network. A real-life analogy of this is that only a few of your lottery tickets will be worth anything but you need to buy a lot in order to find them.
There is a connection to the original rationale behind dropout: Train many networks 'in parallel' and some of the time you will be training the only the golden ticket network.