How does the test error pattern depend on the regularizer function?

Question

This question is regarding the role of regularizer in an objective function.

Given a loss function $f(x)$, a regularizer function $r(x)$, and $\lambda$ being a trade-off function, our aim is to $\min f(x) + \lambda*r(x)$. I'm able to understand that as the value of $\lambda$ increases our training error increases.

I wanted to know how the test error behaves as $\lambda$ increases. From "Pattern Recognition and Machine Learning" by Bishop, I came to know that "the test error initially decreases and then starts increasing". Consequently, we choose the $\lambda$ which has the lowest validation error.

Does the same pattern (decrease and then increase) repeat for any given regularizer function? Or, does this pattern depends on the regularizer function which we use?

Answer 1

Yes-dependent on where the optimal cross-validated value of $\lambda$ lies-which is dependent on the true joint distribution that generates the features and the response (interpret as problem dependent) and the class of functions $\Omega$ such that $f(.) \subseteq \Omega$ (interpret as the kind of model that is being used to fit). As you move from underfitting to the optimal point, the test-error and training-error continue to decrease. Then you reach the optimal $\lambda^*$, after which the Test-error begins to increase, while the training-error continues to decrease- indicating an over-fitting - that is not generalizing the model enough, so as to perform optimally on out-of-sample (unseen) data.