What methods can be used to pick the optimal kernel smoothing width parameter

Question

Kernel smoothers for a function f(x) usually have a parameter which control the width of the region which is used to smooth the value of the function, say at a point $f(x_0)$. For example, the Gaussian kernel

$$ K_\sigma(x) = e^{-\frac{{ \left( {x - x_0 } \right)^2 }}{2\sigma ^2 }} $$

has width parameter $\sigma$ which is the standard deviation.

A choice of greater width increases bias but reduces variance with respect to the variance of the function values surrounding $f(x_0)$, where as a choice of smaller width reduces bias but increases the variance.

What methods can be used to pick the optimal value for the width parameter when using a kernel smoother?

Answer 1

The simplest two methods to compute the bandwidth for kernel smoothing are the following.

• Silverman's rule from B.W. Silverman. "Density Estimation for Statistics and Data Analysis". Chapman and Hall/CRC, 1986. $$ h = \left(\frac{R(k)}{\sigma_k^4}\frac{8\sqrt{\pi}}{3}\right)^{\frac{1}{5}} \hat{\sigma}_n n^{-\frac{1}{5}}. $$ where $\hat{\sigma}_n$ is the sample standard deviation, $\sigma_k^4$ is the standard deviation of the kernel $k$, $R(k)$ is the rugosity of the kernel. This is based on the assumption that the distribution is gaussian. The previous equation comes from computing the rugosity of the second derivative of the gaussian distribution.
• In M.P. Wand and M.C. Jones. "Kernel Smoothing". Chapman and Hall/CRC, 1994, the sample standard deviation is replaced with the robust estimator $$ \hat{\sigma}_{IQR} = \frac{IQR}{\Phi^{-1}(0.75) -\Phi^{-1}(0.25).} $$ where $\Phi^{-1}$ is the quantile of the gaussian standard distribution and $IQR$ is the interquartile range : $$ IQR = \hat{Q}_3 - \hat{Q}_1 $$ where
  $$ \hat{Q}_3 = q(0.75), \qquad \hat{Q}_1 = q(0.25), $$ are the empirical quantiles of level 0.25 and 0.75.

Chapter 3 of (Wand, Jones, 1994) is on this topic.

An interesting method is the "solve-the-equation" rule with two stages due to Sheather and Jones, 1991 (see page 74 in (Wand, Jones, 1994).

Answer 2

A variety of methods can be used and some of them are specific to your application (for instance you might be looking for some misclassification rate of a classifier using the smoothed estimated). Nevertheless most of the time the methodology employed is CV (cross-validation) or GCV (generalized cross-validation). See Generalized Cross-Validation as a Method for Choosing a Good Ridge Parameter (1979) by Golub et al for a first read.

You are essentially looking at hyper-parameter selection problem and this is far from solved. See for example the paper on On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation (2010) by Cawley and Talbot for a fresher look on the subject.