4
$\begingroup$

Working on probabilistic outputs of kernel methods I found the formulation of the SVM as a Penalized Method using the Binomial Deviance (described for example in "The Elements of Statistical Learning Theory, 2nd edition" by Hastie et al., pp 426--430). The model is called "Kernel Logistic Regression (KLR)". References on "The well studied KLR" lead to literature on smoothing splines, but KLR is never mentioned there.

What is the connection between KLR and Smoothing Splines?

Thank you in advance for your answer!

$\endgroup$
2
  • $\begingroup$ what kind of connection you are looking for? KLR is for classification, smoothing splines are for non-linear approximation of conditional expectation. The way I see they are non-linear extensions of logistic and ordinary regressions respectively. $\endgroup$
    – mpiktas
    Commented Dec 30, 2010 at 20:15
  • $\begingroup$ KLR seems to be coming out of nowhere: I found the name KLR to be mentioned first around 2000 (for example in "kernel logistic regression and the import vector machine", Zhu and Hastie, 2001). But they say in there it is well known and reference smoothing spline literature. My Question is: When and where was KLR first introduced and why do they always reference smoothing-splines? $\endgroup$
    – RichardN
    Commented Dec 30, 2010 at 21:15

1 Answer 1

3
$\begingroup$

There is a book on "Nonparametric Regression and Generalized Linear Models: A Roughness Penalty Approach" by Green and Silverman, that is probably a good start, but my copy is in my office, so I can't get it until the new year. Essentially IIRC, the link between kernel methods and smoothing splines is that a regularisation term is used that penalises particular properties of the function implemented by the model, commonly (as the name suggests) the roughness (as measured by the second derivative or curvature). For kernel methods, the regularisation operator depends on the choice of kernel, but the regularisation operator is not dependent on the particular sample of data, which is the connection with splines rather than other non-parametric models. Smoothing splines can be used with more or less any loss function (including the logistic loss), just as kernel methods can use more or less any (convex) loss. Often the squared error or hinge losses are not the best ones, but they get most of the attention, and you can sometimes incorporate useful expert knowledge about the task via the loss just as in GLMs. Hopefully I can give a better answer once I have a chance to refer to my books!

P.S. there is a paper on regularisation networks by Poggio and Girosi that may well be quite relevant as well.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.