Does pruning Deep Neural Networks at initialization even make sense? I recently started exploring pruning methods for Deep Neural Networks and stumbled on some interesting papers suggesting algorithms for unstructured pruning at initialization (e.g. SNIP), i.e. removing single weights (elements in the connection matrices) instead of full neurons (rows/columns of the matrix, depending on notation).
Most research papers state that pruning leads to reductions in training time and memory requirements, which I could not verify in PyTorch. I tried using a (very basic) magnitude based pruning approach at initialization to verify the assumptions that the training and inference time is reduced via pruning. However, both increase significantly when pruning is applied. Follow the link to see a minimal working example as reference.
Pruning is implemented via a binary mask that is element-wise applied in each forward pass (the pruned weights appear to still be updated in the backward pass). The same appears to be the case in Tensorflow. Naturally, adding a matrix multiplication per layer leads to an increase in computational time, which I also observed in the notebook provided above. When reading the literature, I expected that pruning makes use of the resulting sparsity structure in some way, but it appears as if the necessary implementations are still missing.
My question now is: Are the stated gains in efficiency from the papers purely theoretical (for now)? Or is there a way to actually use pruning at initialization to decrease the training time and subsequently the inference time?
Thanks in advance!
 A: Clearly, the problem is how you technically implemented your pruning, not the pruning.
If you did pruning of neurons, you need to really remap everything to smaller dimensions (i.e. you remove the removed neurons and remap all the indices, which is tedious but obvious how it would be done), then computation should be faster.
When you prune weights (instead of neurons), your weight matrix becomes more sparse. As far as I am aware, sparse matrix operations should speed things up. By really setting the weights to zero in weight matrix (to make the matrix partially sparse) instead of matrix-multiplying on the mask every time, you should be able to realize these benefits (at the very least there should be no way this is meaningfully slower than without pruning).
However, note that there are whole packages written to do pruning (e.g. this one) so I would check whether they have implemented what you want/need.
Pruning techniques that aim to optimize actual training/inference speed on real existing devices (or even co-optimizing hardware and neural network) as opposed to just reducing the number of model parameters are areas of ongoing research.
