If I've trained a DNN with out any regularization methods (e.g. weight decay, dropout etc.) and reached a good training error, can I somehow take that learned net and fine tune it with regularization?

The idea behind the question is, that usually when I want to train a DNN with a fixed architecture, I start with no regularization, and try to find optimal hyper-parameters (learning rate, momentum, etc.) for solving the optimization problem of reaching the best training error. Obviously this means overfitting and a bad test error, and so I after finding the best values, I fix the hyper-parameters I've found, and now search over the hyper-parameters that control the regularization (weight decay, dropout rate, etc.), but I have to train the net from random state. If I could take a net that overfit as the initial values, and continue the training with regularization, I could save a lot of computing resources wasted otherwise.

My intuition tells me that it shouldn't be possible, and that the local minima of a net without regularization, and minima of nets with regularization are in different "valleys" of the solution landscape, and so can't be reached without using a large learning rate to skip over the ridges separating them which effectively means destroying the previously found solution. Despite that, I wonder if anyone has tried to tackle this problem, or if maybe using a slowly increasing regularization term would enable such a solution.

  • $\begingroup$ You might look into "regularization path" methods for generalized linear models. In those cases, they actually typically start with a very high regularization penalty, and then decrease it. In that case the model is convex, though, so the issues of re-optimizing are much simpler. $\endgroup$ – Dougal Nov 28 '15 at 17:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.