2
$\begingroup$

Equation 5.30 of The Elements of Statistical Learning states that the penalized log-likelihood for a nonparametric logistic regression is:

\begin{align} l(f;\lambda) &= \sum_{i=1}^{N}\big[y_i\log p(x_i) +(1-y_i)\log(1-p(x_i))\big] - \frac12\lambda \int{f''(t)}^2dt \\ &= \sum_{i=1}^{N}\big[y_if(x_i) - \log(1+e^{f(x_i)}\big] - \frac12\lambda \int{f''(t)}^2dt \end{align}

I've read through Negative binomial log-likelihood in penalized regression and What is penalized logistic regression but I haven't come across this formulation yet.

I'm familiar with LASSO and ridge, but the penalized parameter is a vector of $\beta$ coefficients. In this non-parametric scheme, can someone explain why we penalize $\int{f''(t)}^2dt$ instead? I understand that for the non-parametric case, $f(x)$ acts similar to $\beta$ in the parametric case. But it's not obvious to me why this is how it's formulated. Note that

$$f(x) = \log\frac{P(Y = 1|X=x)}{P(Y=0|X=x)}$$

$\endgroup$
  • $\begingroup$ penalised just means adding a penalty term (which is pretty broad). But the link you refer to refers back to section 5.4 smoothing splines - adding a penalty to enforce smoothness $\endgroup$ – seanv507 Dec 27 '16 at 20:56
  • $\begingroup$ Right, I understand that it's a penalty term. But why that specific form (integral of square of second derivative)? $\endgroup$ – ilanman Dec 28 '16 at 13:38
  • $\begingroup$ en.wikipedia.org/wiki/Smoothing_spline $\endgroup$ – seanv507 Dec 28 '16 at 13:58
  • $\begingroup$ Of course Wikipedia! So, like most things, this penalty is chosen for convenience/out of tradition. Thanks for the link. $\endgroup$ – ilanman Dec 28 '16 at 14:03
0
$\begingroup$

Per @seanv507's link to Wikipedia:

This formulation is based on the class of twice differentiable functions, and

The roughness penalty based on the second derivative is the most common in modern statistics literature, although the method can easily be adapted to penalties based on other derivatives.

This answers my question.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.