2
$\begingroup$

Equation 5.30 of The Elements of Statistical Learning states that the penalized log-likelihood for a nonparametric logistic regression is:

\begin{align} l(f;\lambda) &= \sum_{i=1}^{N}\big[y_i\log p(x_i) +(1-y_i)\log(1-p(x_i))\big] - \frac12\lambda \int{f''(t)}^2dt \\ &= \sum_{i=1}^{N}\big[y_if(x_i) - \log(1+e^{f(x_i)}\big] - \frac12\lambda \int{f''(t)}^2dt \end{align}

I've read through Negative binomial log-likelihood in penalized regression and What is penalized logistic regression but I haven't come across this formulation yet.

I'm familiar with LASSO and ridge, but the penalized parameter is a vector of $\beta$ coefficients. In this non-parametric scheme, can someone explain why we penalize $\int{f''(t)}^2dt$ instead? I understand that for the non-parametric case, $f(x)$ acts similar to $\beta$ in the parametric case. But it's not obvious to me why this is how it's formulated. Note that

$$f(x) = \log\frac{P(Y = 1|X=x)}{P(Y=0|X=x)}$$

Improve this question
$\endgroup$
4
  • $\begingroup$ penalised just means adding a penalty term (which is pretty broad). But the link you refer to refers back to section 5.4 smoothing splines - adding a penalty to enforce smoothness $\endgroup$
    – seanv507
    Dec 27, 2016 at 20:56
  • $\begingroup$ Right, I understand that it's a penalty term. But why that specific form (integral of square of second derivative)? $\endgroup$
    – ilanman
    Dec 28, 2016 at 13:38
  • $\begingroup$ en.wikipedia.org/wiki/Smoothing_spline $\endgroup$
    – seanv507
    Dec 28, 2016 at 13:58
  • $\begingroup$ Of course Wikipedia! So, like most things, this penalty is chosen for convenience/out of tradition. Thanks for the link. $\endgroup$
    – ilanman
    Dec 28, 2016 at 14:03

1 Answer 1

Reset to default
1
$\begingroup$

Per @seanv507's link to Wikipedia:

This formulation is based on the class of twice differentiable functions, and

The roughness penalty based on the second derivative is the most common in modern statistics literature, although the method can easily be adapted to penalties based on other derivatives.

This answers my question.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.