Ridge regression's objective function: $$ L(w) = \underbrace{\|y - Xw\|^2}_\text{data term} + \underbrace{\lambda\|w\|^2}_\text{smoothness term} $$
I am trying to understand the regularization term, $\lambda\|w\|^2$. My questions are:
What does smoothness mean here?
I checked the definition of smooth in Wolfram, but it seems not right in here.
A smooth function is a function that has continuous derivatives up to some desired order over some domain.
I read a document explaining the smoothness term.
A very common assumption is that the underlying function is likely to be smooth, for example, having small derivatives. Smoothness distinguishes the examples in Figure 2. There is also a practical reason to prefer smoothness, in that assuming smoothness reduces model complexity:
I have difficulty understanding above:
the underlying function is smooth will have small derivatives
smoothness reduces model complexity.
My counterexample is: $$ f(x) = w_0 + w_1x + w_2x^2 + w_3x^3 $$
with $w = [0.5, 0.7, 0.3, 0.4]$ , or $w = [5, 7, 3, 4]$, they are both function of $C^\infty$
I know I must be making mistakes somewhere. Please help me to correctly understand it. Thank you.