11
$\begingroup$

Recently I have used Platt's scaling of SVM-outputs to estimate probabilities of default-events. More direct alternatives seem to be "Kernel logistic Regression" (KLR) and the related "Import Vector Machine".

Can anyone tell which kernel method giving probability-outputs is currently state of the art? Does an R-implementation of KLR exist?

Thank you very much for your help!

Improve this question
$\endgroup$
1
  • $\begingroup$ (+1) A very interesting question ... $\endgroup$
    – steffen
    Commented Dec 28, 2010 at 21:51

2 Answers 2

Reset to default
8
$\begingroup$

Gaussian process classification (using Expectation Propagation) is probably the state-of-the-art in machine learning. There is an excellent book by Rasmussen and Williams (downloadable for free), the website for which has a very good MATLAB implementation. More software, books, papers etc. here. However, in practice, KLR will probably work just as well for most problems, the major difficulty is in selecting the kernel and regularisation parameters, which is probably best done by cross-validation, although leave-one-out cross-validation can be approximated very efficiently, see Cawley and Talbot (2008).

Improve this answer
$\endgroup$
4
  • $\begingroup$ I should add, don't use implementations based on the Laplace approximation - the posterior is highly skewed, and a symmetric approximation centered on the mode generally wont work very well. $\endgroup$
    – Dikran Marsupial
    Commented Dec 29, 2010 at 21:16
  • $\begingroup$ Thank you Dikran! Could you explain to me the relation of KLR und Kernel smoothing? The KLR-model is built similar to the svm [loss + penalty]-formulation and solved via gradient descent. But the same time references (e.g. in "Kernel Logistic Regression and the Import Vector Machine", Zhu and Hastie 2005) on KLR go to the smoothing-literature (e.g. "Generalized Additive Models", Hastie and Tibshirani 1990). $\endgroup$
    – RichardN
    Commented Dec 30, 2010 at 10:59
  • $\begingroup$ I am not that familiar with the smoothing literature, but kernel models are closely related to spline smoothing. I think the best place to look would be the publications by Grace Wahba (stat.wisc.edu/~wahba), whos work spans both smoothing and kernel methods. $\endgroup$
    – Dikran Marsupial
    Commented Dec 30, 2010 at 11:43
  • $\begingroup$ Thanks, i will have a closer look at wahba's publications. Can you recommend an implementation of KLR, at best in R? $\endgroup$
    – RichardN
    Commented Dec 30, 2010 at 12:03
1
$\begingroup$

I guess you know that the kernel for logistic regression is a non parametric one, so first of all you have that restriction.

Regarding the R package the one I know and works pretty well is np: Nonparametric kernel smoothing methods for mixed data types

This package provides a variety of nonparametric (and semiparametric) kernel methods that seamlessly handle a mix of continuous, unordered, and ordered factor data types.

Regarding the state of the art kernell I can recomend to experiment with the ones described in this paper from 2009. Read it carefully to choose the one that is best and more actual for you.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Hey Mariana, thanks for your answer, but we had a misunderstanding: I by "kernel methods" mean methods such as the Support vector machine using the "kernel trick", not kernel smoothing methods. $\endgroup$
    – RichardN
    Commented Dec 29, 2010 at 10:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.