Help with kernel regression calculation

Question

(This is a really basic question, so I appreciate your patience.)

I'm trying to do a Nadaraya-Watson kernel regression on a set of $(x,y)$ data points to predict the value at a particular $x$.

I have the formula from wikipedia: $\widehat{m}_h(x)=\frac{\sum_{i=1}^n K_h(x-x_i) y_i}{\sum_{i=1}^nK_h(x-x_i)} $

Where the kernel looks like: $K(u) = \frac{3}{4}(1-u^2)\ 1_{(|u|\leq1)}$

What I don't understand is how to use the formula for $K(u)$. I understand that there is a bandwidth $h$, but where does it go? Do I need to scale the input by $h$, or is the sum in the formula for $\widehat{m}_h(x)$ get weighted by h? Or something else?

Answer 1

Note that $h$ is just a scale parameter: $K_h(u) = \frac{1}{h}K(\frac{u}{h})$.

So for the Epanechnikov kernel

Consider what's going on at one particular value of $x$, $x_0$, say: you're simply taking a weighted average of the $y$'s:

$\hat{y}_0 = \bar{y}^{(w)}=\frac{\sum_i w_iy_i}{\sum_i w_i}$

The weights are set up so we weight the observations down as we move away from $x_0$, where the weight on $y_i$ decreases down to 0 when $x_i$ is $h$ away from $x_0$; the weights are given by the kernel function.

So for example if we take the mcycle data from the R package MASS and choose $h=2$, then at $x_0=20$ we can calculate $w_i=\frac12 K((x_i-20)/2)$:

The points in this region with the corresponding weights are:

     times accel     K2(x-20)
52   17.8  -99.1     0.00000
53   17.8 -104.4     0.00000
54   18.6 -112.5     0.19125
55   18.6  -50.8     0.19125
56   19.2 -123.1     0.31500
57   19.4  -85.6     0.34125
58   19.4  -72.3     0.34125
59   19.6 -127.2     0.36000
60   20.2 -123.1     0.37125
61   20.4 -117.9     0.36000
62   21.2 -134.0     0.24000
63   21.4 -101.9     0.19125
64   21.8 -108.4     0.07125
65   22.0 -123.1     0.00000

Note that the $K_2$ values -- the weights -- are computed from times but used as weights in the average of accel. The points that are included in the average are shaded according to weight (the deep blue ones in the middle get more weight than the less heavily shaded points nearer to the bound of the kernel near 18 and 22)

If we apply those weights* on observations with $x$ between 18 and 22 (since outside that region the weighted points contribute nothing), and calculate the weighted average, that will be the fitted value at 20. In this case it comes out to -106.67 (marked in blue on the plot).

* (these don't sum to 1, which is why we divide by the sum of the weights)

We can then repeat this kind of calculation every place we want to have an estimate of the smoothed function.

Answer 2

There's a hint in said Wikipedia article that leads to the correct solution. Under the Derivation heading of the Nadaraya-Watson estimator, it links to kernel density estimate.

When performing a kernel density estimate of univariate data, we have that
$\hat{f}_h(x) = \frac{1}{n}\sum_{i=1}^n K_h (x - x_i) = \frac{1}{nh} \sum_{i=1}^n K\Big(\frac{x-x_i}{h}\Big)$

(from the linked page on KDEs.)

Hopefully you can take it from here.