Is there a non-boostrap way to estimate confidence intervals for Kernel regression predictions? Simple problem of estimating:
$$ y = f(x) + \epsilon $$
Where I use your standard Nadaraya-Watson Regression to guess $f(x)$. This is relatively fast and works well even in an online setting.
Now I use the regression to predict $y'$ at $x'$. However in my use case I'd really like to get a confidence interval around my predicted $\hat y'$.
Usually I'd solve this by bootstrap, but for my application it is just too computationally expensive (it is an online problem where a stream of data keeps coming in and I just can't stop to do bootstrap at each step).
Is there an approximative way to get a prediction error interval? It seems like I should be able to say "this prediction is far away from where I have the data so my confidence is low".
 Any help?
 A: Under non-fixed-design $x$'s, [Härdle][1] (p136) gives an asymptotic distributional approximation relating to the Nadaraya-Watson estimator $\hat{m}(x_j)$:
$(nh)^\frac12 \frac{\hat{m}_h(x_j)-m(x_j)}{V(x_j)^\frac12} \stackrel{.}{\sim} N(B(x_j),1)$
where $V(x_j) = \sigma^2(x_j)||K||_2^2/f(x_j)$ and $B(x_j) = \mu_2(K)[m''(x_j)+2m'(x_j)f'(x_j)/f(x_j)]$,
with $\sigma^2(x_j)$ being the conditional noise variance, $f(x_j)$ being the density of $X$ at $x_j$, $||K||_2^2 = \int K^2(u) du$, $\mu_2(K) = \int u^2 K(u) du$.
(Actually he gives the multivariate distribution over $k$ different locations but since you seem to only need it pointwise, I believe the univariate case suffices)
Note that $V$ is the variance of the estimate of $m$. The approximate predictive variance would then add the observation noise variance $\sigma^2(x_j)$ to give $V^p(x_j)=\sigma^2(x_j)+V(x_j)$. The MSPE at $x_j$ is the sum of the variance and the square of the bias, so an asymptotic approximation for that is $V^p(x_j)+B(x_j)^2$.
The bias term, $B$ involves derivatives of both $m$ and $f$ and so is somewhat complicated to evaluate/estimate (but should be doable, nonetheless).
Some people make the assumption that $B$ is small relative to the variance and ignore it; this would give a lower bound on the prediction error variance.
Härdle gives estimators for $\sigma^2(x)$ and $f(x)$ based on NW-kernel regression from the squared error and kernel density estimation respectively (all using the same bandwidth) and he gives tables for $\mu_2(K)$ and $||K||_2^2$ on p220 (the kernels themselves are defined on p45).
For the Gaussian case
$\mu_2(K)=1$ and $||K||_2^2=\frac{1}{2\sqrt{\pi}}$.
[1]: Härdle, W. (1991),
Smoothing Techniques - With Implementation in S,
Springer Series in Statistics,
Springer-Verlag New York
