I'm trying to understand how to choose an appropriate bandwidth for kernel regression. Note that this is NOT about kernel density estimation (unless someone can convince me that the same techniques can be used).

Here's my thinking on this: The bandwidth should be allowed to decrease as:

  • 1) more data is gathered.
  • 2) there are known variations/oscillations in the data of a certain size (e.g. a sine wave of an approximate frequency of 0.5 units of the predictor variable.)

These concepts are the same whether I'm talking about LOWESS or Nadaraya -Watson - they both use a bandwidth during estimation.

I'm aware of Silverman's rule for KDE, but is there an equivalent for kernel regression that captures my intuition above?

Of course I can determine it experimentally using brute force grid search method, but this is very computationally expensive, and won't scale when there are more than 2 dimensions. Thank you.


2 Answers 2


I would recommend you to read this beautiful article by Racine and Li published in the Journal of Econometrics in 2004. They develop a framework to estimate regression functions nonparametrically using kernel methods, with mixed types of covariates (categorical or continuous regressors). Among other results, they show consistency of the cross-validated estimates. This is a classical article in the nonparametric econometrics literature.

The main method for choosing bandwidth parameters is, undoubtfully, the cross-validation procedure. However, other methods exist such as bootstraping (a quick google search gives: a PhD thesis about choosing bandwidths for np kernel regression.

If you have a large sample, leave-one-out CV may not be the way to go for computational reasons. Moreover, if the data consist of time-series, the CV method may not be valid anymore. What you can do is proceed with hold-out validation. Assume you have $T$ observations.

  1. Split the sample into two parts: the estimation sample (observation 1 to $T-k$) and a hold-out sample (obs $T-k+1$ to $T$).
  2. Compute the estimator using the estimation sample (first $T-k$ obs) as a function of $h$.
  3. Compute the out-of-sample prediction for the hold-out sample (last $k$ obs) as a function of $h$.
  4. Minimize the squared prediction error with respect to $h$.
  5. Recompute $h$ using the same procedure as new data come in.

Also, if the Nadaraya-Watson estimator is indeed a np kernel estimator, this is not the case for Lowess, which is a local polynomial regression method. You could also fit your regression function using the Sieves (i.e. through a basis expansion of the function) based on wavelets for example given the structure of your data.

Finally, the np kernel estimation of densities is extremely similar to that of the conditional mean, which is what you have in mind when you talk about 'regression'. A np kernel regression considers estimating $E(Y|X)$, where $Y$ is the dependent variable and $X$ is a (hopefully) exogenous predictor. Replacing Y by $I(Y\leq y)$ - where $I$ denotes the indicator function that equates $1$ when the event inside the brackets occurs - gives the conditional mean expression $E(I(Y\leq y)|X)$. Now, run a bunch of np kernel regression on $E(I(Y\leq y)|X)$ for various values of $y$. This will provide an estimate the conditional cumulative distribution of $Y$ given $X$. Now take a derivative with respect to $y$, you have a density. So what's the difference after all? Just a matter of choosing the dependent variable. The method is the same, unless you want to re-scale and impose restrictions on the CDF maybe...

  • $\begingroup$ Is it reasonable to say that the time complexity for the leave-one-out method is O(n^3 * log(n))? (* n^2 to compute one regression line with a specific bandwidth and set of data points, * n to compute a separate regression line for each data point left out, and * log(n) for gradient descent to the optimal bandwidth? $\endgroup$ Commented Aug 25, 2021 at 16:17

Here is what I could find in study notes from my econometrics class:

Average squared error (ASE) is given as: $$ASE = \frac{1}{n} \sum_{j=1}^n[\widehat{m}_h(X_j)-{m}_h(X_j)]^2 w(X_j)$$ where $\widehat{m}_h(X_j)$ is the fitted value from the kernel regression, $w(x)= \sum_{i=1}^n \frac{K(\frac{x-X_i}{h})}{\sum_{j=1}^nK(\frac{x-X_j}{h})}$ with K being the chosen kernel function and $h$ is the bandwith. The mean average squared error (MASE) is as follows: $$MASE=E[ASE\vert X_1=x_1,...,X_n=x_n]$$ The suggested approach for finding the optimal (in a minimum MASE sense) bandwith $h$ is cross validation: $$CV(h)=\frac{1}{n}\sum_{i=1}^n[Y_i-\widehat{m}_{h,i-1}(X_j)]^2w(X_i)$$ where this must be minimized for $h$. It can be shown that MSE at a given x is a function of both variance and bias, and depends on the number of data points being 'close' and the curvature of the function.

  • $\begingroup$ So, we are assuming that the bandwidth must be constant throughout the entire regression. Can we vary the bandwidth with the number of data points, or the sample variance of the data? $\endgroup$
    – makansij
    Commented Apr 13, 2016 at 15:07
  • 1
    $\begingroup$ In the above setup $h$ will be fixed. Theoeretically optimal $h$ (minimizing MSE, which cannot be obtained in practice) would depend upon x (so on the parameters you mention). Cross-Validation is a method to get a global $h$ from data and it minimizes MASE. I would assume there are other approaches, perhaps with varying bandwith, but I am not aware of any. Also note that the explanation above is for a regression with one explanatory variable. $\endgroup$ Commented Apr 13, 2016 at 15:25
  • $\begingroup$ One more hint. I think my professor took the formulas from this book, which you should be able to access for free: researchgate.net/profile/Marlene_Mueller/publication/… $\endgroup$ Commented Apr 13, 2016 at 15:38
  • $\begingroup$ The link that you sent, for some reason the pictures don't show up. I don't understand what $w(x)$ means. It seems like it would always be 1. You can factor out the denominator because it doesn't depend on $i$, and then when you sum those values in the numerator, they are the same as the denominator? $\endgroup$
    – makansij
    Commented Apr 16, 2016 at 19:56
  • $\begingroup$ Basically this is a weight, the formula is shown above. The denominator sums over all observations but $X_i$ $\endgroup$ Commented Apr 16, 2016 at 19:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.