This question is regarding the role of regularizer in an objective function.
Given a loss function $f(x)$, a regularizer function $r(x)$, and $\lambda$ being a trade-off function, our aim is to $\min f(x) + \lambda*r(x)$. I'm able to understand that as the value of $\lambda$ increases our training error increases.
I wanted to know how the test error behaves as $\lambda$ increases. From "Pattern Recognition and Machine Learning" by Bishop, I came to know that "the test error initially decreases and then starts increasing". Consequently, we choose the $\lambda$ which has the lowest validation error.
Does the same pattern (decrease and then increase) repeat for any given regularizer function? Or, does this pattern depends on the regularizer function which we use?