6
$\begingroup$

If we use a way to control the complexity of our regression model by adding a penalty terms to the error: $\hat{E} = E + \lambda \Omega$ and use as regularization term omega for example a penalty for large weights $\Omega = \frac{||w||^2}{2}$ with $w$ being the vector of weight, why do we need to normalize our data before?

$\endgroup$
0

1 Answer 1

7
$\begingroup$

The reason to normalise your variables beforehand is to ensure that the regularisation term $\lambda$ regularises/affects the variable involved in a (somewhat) similar manner. A very interesting thread touching on this issue appeared is here where the regularisation was imposed to normalised and unnormalised data and unsurprisingly the results where quite different.

In brief when you impose something like : $(X^TX + \lambda \Omega)^{-1}$ instead of the usual $(X^TX)^{-1}$ you want the effect of $\lambda$ to even across all variables constituting $X$. If the variables are of significantly different scales (think $x_1$ as being of the order $1e1$ and $x_2$ being of the order $1e{5}$, so for example age in years and weight in grams respectively) regularizing (roughly speaking accepting some degree of error in the measurements) the two by the same magnitude is nonsensical. The well-referenced Wikipedia lemmas on Tikhonov regularization and Matrix regularization are good places to follow up on this in more theoretical manner if you are interested.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.