I have a predictive model with a relative small number of model parameters (only 6). When I train the model on the training set and than validate the model on the validation set I have a strong indication of an over-fit (out-of-sample error is significantly larger than the in-sample error).

So, to solve the problem I can try to regularize by implying penalties on large values of the model parameters. However, in this particular case, I do not see a reason why zero values of the model parameters should provide a simple model. For example I have an angle as a model parameter and I do not see why zero degree is "better" than, let's say, 30 degree.

How to do a regularization in this case?

  • 3
    $\begingroup$ The regularization, or shrinkage towards zero applies to the coefficients, not the actual values of the explanatory variable. So the original values of the angles you are referring to would remain the same. Just the coefficient for the effect of a different angle is shrunk. $\endgroup$ – Frans Rodenburg Oct 28 '17 at 14:09
  • 1
    $\begingroup$ Just a note that: "out-of-sample error is significantly larger than the in-sample error" is not necessarily a sign of overfitting. $\endgroup$ – Matthew Drury Oct 29 '17 at 19:43

You can regularize relative to any set of parameter values you like. This can always be done by writing appropriate regularization code--but such an effort is usually unnecessary. For models of a response $y$ in terms of variables $X$ that are linear in the sense

$$y = \beta_1 f_1(X) + \beta_2 f_2(X) + \cdots + \beta_p f_p(X) + \text{ error}\tag{1}$$

(for specified functions $f_1, \ldots, f_p$), suppose you consider the "simplest" possible value of parameter $\beta_i$ to be $\alpha_i$. Rewrite model $(1)$ into the algebraically and probabilistically equivalent form

$$y - \alpha_1 f_1(X) - \cdots - \alpha_p f_p(X) = (\beta_1-\alpha_1) f_1(X) + \cdots + (\beta_p-\alpha_p) f_p(X) + \text{ error}.$$

Because you specify the $\alpha_i$, you can compute the left hand side directly as part of data preprocessing. Call its values $y^\prime$. Proceed to fit the model

$$y^\prime = \gamma_1f_1(X) + \gamma_2 f_2(X) + \cdots + \gamma_p f_p(X) + \text{ error},$$

regularizing the coefficients (and likely shrinking them towards zero). Simply set the model $(1)$ estimates to $\hat\beta_i = \hat\gamma_i + \alpha_i$ afterwards. Any estimate that has been shrunk to $\hat\gamma_i=0$ will yield $\hat\beta_i=\alpha_i$, as planned.

Please note that this works in part because subtracting terms like $\alpha_i f_i(X)$ from $y$ leaves the additive error term unaffected. It wouldn't do to, say, divide both sides by such terms, because the error term would be divided by numbers that might vary from one case to the next. That would change the model.


If you have any domain knowledge or ground truth data on the angle parameter, then you can come up with a probability distribution for the model (for example, modeling the angle distribution as a gaussian mixture), and then add $-\log P(\text{angle})$ as a penalty. The usual L1 or L2 penalty is a special case of this.

You could also try adding noise to the training data as a form of regularization, since a set of parameters which are more robust are less likely to overfit.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.