If I've trained a DNN with out any regularization methods (e.g. weight decay, dropout etc.) and reached a good training error, can I somehow take that learned net and fine tune it with regularization?
The idea behind the question is, that usually when I want to train a DNN with a fixed architecture, I start with no regularization, and try to find optimal hyper-parameters (learning rate, momentum, etc.) for solving the optimization problem of reaching the best training error. Obviously this means overfitting and a bad test error, and so I after finding the best values, I fix the hyper-parameters I've found, and now search over the hyper-parameters that control the regularization (weight decay, dropout rate, etc.), but I have to train the net from random state. If I could take a net that overfit as the initial values, and continue the training with regularization, I could save a lot of computing resources wasted otherwise.
My intuition tells me that it shouldn't be possible, and that the local minima of a net without regularization, and minima of nets with regularization are in different "valleys" of the solution landscape, and so can't be reached without using a large learning rate to skip over the ridges separating them which effectively means destroying the previously found solution. Despite that, I wonder if anyone has tried to tackle this problem, or if maybe using a slowly increasing regularization term would enable such a solution.