Background knowledge I knew about regularization in machine learning: A regularization term is added to the objective function to prevent overfitting. Usually, one would like to restrict the variance of the model and hence adding constraints to the model such that the parameters $\theta$ that will be learned will not grow too much. The following post explains well:
However, I am a bit confused with the newly added parameter, i.e. Lagrange multiplier $\lambda$. The question is, is $\lambda$ learned just like $\theta$ (original parameters of the model that you don't want wild growth)? Or, one simply set a $\lambda$ to begin with, while the objective function is the one composed of the original function and the Lagrange multiplier terms?
I have this question because I never work through a regularization myself and from the first post of the above links, it said "Hence reducing $\lambda$ generates a sequence of hypothesis spaces of increasing complexity." It sounds like one can choose a $\lambda$ and hence choose a model complexity (the smaller the $\lambda$ is chosen, the less overfitting possibility). Another question is, if this is true, i.e. one can set $\lambda$ at one's wish, how do I understand the effect of the size of $\lambda$ on the overfitting issue?
Thank you guys!