Are Lagrange multipliers of regularization terms learned or set?

Question

Background knowledge I knew about regularization in machine learning: A regularization term is added to the objective function to prevent overfitting. Usually, one would like to restrict the variance of the model and hence adding constraints to the model such that the parameters $\theta$ that will be learned will not grow too much. The following post explains well:

What is the connection between regularization and the method of lagrange multipliers ?

https://www.quora.com/Is-Regularization-in-Machine-Learning-Neural-Network-just-another-way-of-implementing-Lagrange-multiplier-1

However, I am a bit confused with the newly added parameter, i.e. Lagrange multiplier $\lambda$. The question is, is $\lambda$ learned just like $\theta$ (original parameters of the model that you don't want wild growth)? Or, one simply set a $\lambda$ to begin with, while the objective function is the one composed of the original function and the Lagrange multiplier terms?

I have this question because I never work through a regularization myself and from the first post of the above links, it said "Hence reducing $\lambda$ generates a sequence of hypothesis spaces of increasing complexity." It sounds like one can choose a $\lambda$ and hence choose a model complexity (the smaller the $\lambda$ is chosen, the less overfitting possibility). Another question is, if this is true, i.e. one can set $\lambda$ at one's wish, how do I understand the effect of the size of $\lambda$ on the overfitting issue?

Thank you guys!

Answer 1

$\lambda$ is a hyper-parameter. Typically user must set the hyper-parameter. There are some ways of automatically selecting a hyper-parameter by optimizing an auxiliary function, for example as done in bayesian optimization (a friendly intro is here).

For neural networks, I like the strategy for setting $\lambda$ as discussed by Andrej Karpathy's, which is you first set it to 0, and let the model overfit, and then slowly increase it until you have an acceptable generalization error.

So yes, you can achieve different results by setting $\lambda$ to a fixed value and fitting your model.

Another question is, if this is true, i.e. one can set $\lambda$ at one's wish, how do I understand the effect of the size of $\lambda$ on the overfitting issue?

With $\lambda=0$, there is no regularization of the parameters, i.e. we are not imposing any constraints on the parameters. They are overly 'flexible' in how they want to arrange themselves to fit a function. To give you an analogy, let's say you're trying to train your dog to be able to jump high. One aspect of this training is diet. Initially you impose no constraints on the diet, and your dog eats whatever it wants, however much it wants. Naturally this can affect performance.

As you start imposing constraints (e.g. using $L_1$ or $L_2$ regularization) and how strictly you enforce it (setting the weight of the regularization coefficient, which is $\lambda$), you take away the 'flexibility' enjoyed by the parameters in how they want to arrange themselves in the parameters space to fit a function. So there are two aspects really: 1) what kind of rules or constraints you impose ($L_1$, $L_2$ etc), and how strongly you enforce it (magnitude of $\lambda$). Both of these choices influence the outcome. This is like first deciding what kind of diet your dog can have access to (e.g. protein-rich or fat-rich), and secondly how much of the food dog can eat. It is reasonable to think by carefully controlling what and how much your dog eats, it will be able to learn to jump high (the task).

Now let's imagine you're an overly harsh person and you impose a very strict constraint. You say your dog can only eat very little amount of food (very high magnitude of $\lambda$). Beyond a certain restriction, now this is beginning to get counter productive. Your dog is getting too little nutrition to live, let alone to learn how to jump high. At this point, your dog will simply not learn anything, that is, it is underfitting to the task of learning how to jump.

Hope this analogy helps.