Are there any algorithms/methods for taking a trained model and reducing its number of weights with as little negative effect as possible to its final performance?

Say I have a very big (too big) model which contains X weights and I want to cut it down to have 0.9*X weights with as little damage as possible to the final performance (or maybe even to the highest possible gain in some cases).

Weight reduction occurs either by changing the model's basic architecture and removing layers or by reducing feature depth in said layers. Obviously after reduction some fine-tuning of the remaining weights will be required.

  • 2
    $\begingroup$ There indeed is research on that. I think it's called network pruning. An example project: keras-surgeon does just that. $\endgroup$ – Jakub Bartczuk Dec 20 '18 at 16:39
  • $\begingroup$ Is this what regularization does? $\endgroup$ – endolith Dec 21 '18 at 17:14
  • $\begingroup$ In a way, weight regularization does this as part of the training process (due to the regularization, some of the weight matrices are practically equal to zero which results in a simpler model). But I was asking about something different which is called pruning. $\endgroup$ – Mark.F Dec 22 '18 at 8:29

You might wanna check:


And a more recent paper on the topic:


However, I was not able to find an implementation of these two. So you will need to implement it yourself.

| cite | improve this answer | |
  • 3
    $\begingroup$ I think this answer would be improved if you provided complete citations for these articles (because the links can go dead) and included a summary of the main results from these links (because just providing links isn't enormously helpful). $\endgroup$ – Sycorax Dec 20 '18 at 16:49

After reading some of the helpful comments and answers, I've did some focused reading on my own.

As mentioned in other answers, this process is called Pruning and like many other ideas in the neural network area, it is not new. From what I can tell, it originates in LeCun's 1990 paper with the lovely "Optimal Brain Damage" (The paper cites some earlier works on network minimization from the late 80's but I didn't go that far down the rabbit hole). The main idea was to approximate the change in loss caused by removing a feature map and minimize it:

∆C(hi) = |C(D|W0) − C(D|W)|

Where C is the cost function, D is our dataset (of x samples and y labels) and W are the weights of the model (W0 are the original weights). hi is the output produced from parameter i, which can be either a full feature map in convolution layers or a single neuron in dense layers.

More recent works on the subject include:

"2016 - Pruning convolutional neural networks for resource efficient inference"

In this paper they propose the following iterative process for pruning CNNs in a greedy manner:

enter image description here

They present and test several criteria for the pruning process. The first and most natural one to use is the oracle pruning, which desires to minimize the difference in accuracy between the full and pruned models. However it is very costly to compute, requiring ||W0|| evaluations on the training dataset. More heuristic criteria which are much more computationally efficient are:

  • Minimum Weight - Assuming that a convolutional kernel with low L2 norm detects less important features than those with a high norm.
  • Activation - Assuming that an activation value of a feature map is smaller for less impotent features.
  • Information Gain - IG(y|x) = H(x) + H(y) − H(x, y), where H is the entropy.
  • Taylor Expansion - Based on the Taylor expansion, we directly approximate change in the loss function from removing a particular parameter.

2016 - Dynamic Network Surgery for Efficient DNNs Unlike the previous methods which accomplish this task in a greedy way, they incorporate connection splicing into the whole process to avoid incorrect pruning and make it as a continual network maintenance. With this method, without any accuracy loss, they efficiently compress the number of parameters in LeNet-5 and AlexNet by a factor of 108× and 17.7× respectively.

enter image description here

The figures and a much of what I written is based on the original papers. Another useful explanation can be found in the following link: Pruning deep neural networks to make them fast and small.

A good tool for modifying trained Keras models is the Keras-surgeon. It currently enables easy methods to: delete neurons/channels from layers, delete layers, insert layers and replace layers.

I didn't find any methods for the actual pruning process (testing criteria, optimizing etc.)

| cite | improve this answer | |
  • $\begingroup$ There is quite a lot more to network pruning (and generally model compression) than this. I suggest you read openreview.net/forum?id=rJlnB3C5Ym and openaccess.thecvf.com/content_ECCV_2018/papers/… $\endgroup$ – DeltaIV Dec 21 '18 at 16:12
  • $\begingroup$ Thank you. I've only recently been exposed to this area. This was just my accumulated knowledge over 36 hours of reading several sources. You additional papers are appreciated. $\endgroup$ – Mark.F Dec 22 '18 at 8:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.