9
$\begingroup$

Currently learning ridge regression and I was a little confused about the penalisation of more complex models (or the definition of a more complex model).

From what I understand, model complexity doesn't necessarily correlate to the polynomial order. So: $$ 2 + 3+ 4x^2 + 5x^3 + 6x^4$$ is a more complex model than: $$ 5x^5$$

And I know that the point of regularisation is to keep the model complexity low, so say for example we have a 5th order polynomial $$ f(x; w) = w_0 + w_1x + w_2x^2 + w_3x^3 + w_4x^4 + w_5x^5$$

The more parameters that are 0 the better.

But what I don't understand is, if it was the same order polynomial why do lower parameter values get penalised less? So why would:

$$ 2 + 5x + x^3$$ be a less complex model than

$$ 433+ 342x + 323x^3$$ they are both of the same polynomial order, and the parameter values just simply dependent on the data.

Thank you!

$\endgroup$
10
$\begingroup$

the parameter values just simply dependent on the data

This is the key part of your question. This is where you are confused.

Yes, the parameter values depend on the data. But the data are fixed when we fit a model. In other words, we fit a model conditional on the observations. It does not make sense to compare the complexity of different models that were fitted to different datasets.

And in the context of a fixed dataset, a model

$$ 2 + 5x + x^3$$

is indeed closer to the simplest possible model, namely the flat zero model, than

$$ 433+ 342x + 323x^3,$$

and this holds regardless of the scale of your observations.

Incidentally, the intercept ($2$ and $433$ in your example) is frequently not penalized, e.g., in most Lasso formulations, because we are usually good with letting it vary freely to capture the overall average of the observations. In other words, we shrink the model towards the average of the observations, not a complete zero model (where the zero would often be arbitrary). In this sense, a flat $2$ and a flat $433$ model would be considered equally complex.

$\endgroup$
  • 1
    $\begingroup$ The lower magnitude coefficients are farther away from flat zero than the higher coefficients? Is that a typo, or am I misunderstanding why a farther-away-from-constant model is not penalized as much as a closer-to-constant model? $\endgroup$ – R.M. May 5 '19 at 1:45
  • $\begingroup$ Sorry, that was indeed a typo. Let me edit. Thanks for pointing this out! $\endgroup$ – Stephan Kolassa May 5 '19 at 3:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.