2
$\begingroup$

(This is a really basic question, so I appreciate your patience.)

I'm trying to do a Nadaraya-Watson kernel regression on a set of $(x,y)$ data points to predict the value at a particular $x$.

I have the formula from wikipedia: $\widehat{m}_h(x)=\frac{\sum_{i=1}^n K_h(x-x_i) y_i}{\sum_{i=1}^nK_h(x-x_i)} $

Where the kernel looks like: $K(u) = \frac{3}{4}(1-u^2)\ 1_{(|u|\leq1)}$

What I don't understand is how to use the formula for $K(u)$. I understand that there is a bandwidth $h$, but where does it go? Do I need to scale the input by $h$, or is the sum in the formula for $\widehat{m}_h(x)$ get weighted by h? Or something else?

$\endgroup$
3
$\begingroup$

Note that $h$ is just a scale parameter: $K_h(u) = \frac{1}{h}K(\frac{u}{h})$.

So for the Epanechnikov kernel

enter image description here

Consider what's going on at one particular value of $x$, $x_0$, say: you're simply taking a weighted average of the $y$'s:

$\hat{y}_0 = \bar{y}^{(w)}=\frac{\sum_i w_iy_i}{\sum_i w_i}$

The weights are set up so we weight the observations down as we move away from $x_0$, where the weight on $y_i$ decreases down to 0 when $x_i$ is $h$ away from $x_0$; the weights are given by the kernel function.

So for example if we take the mcycle data from the R package MASS and choose $h=2$, then at $x_0=20$ we can calculate $w_i=\frac12 K((x_i-20)/2)$:

![![enter image description here

The points in this region with the corresponding weights are:

     times accel     K2(x-20)
52   17.8  -99.1     0.00000
53   17.8 -104.4     0.00000
54   18.6 -112.5     0.19125
55   18.6  -50.8     0.19125
56   19.2 -123.1     0.31500
57   19.4  -85.6     0.34125
58   19.4  -72.3     0.34125
59   19.6 -127.2     0.36000
60   20.2 -123.1     0.37125
61   20.4 -117.9     0.36000
62   21.2 -134.0     0.24000
63   21.4 -101.9     0.19125
64   21.8 -108.4     0.07125
65   22.0 -123.1     0.00000

Note that the $K_2$ values -- the weights -- are computed from times but used as weights in the average of accel. The points that are included in the average are shaded according to weight (the deep blue ones in the middle get more weight than the less heavily shaded points nearer to the bound of the kernel near 18 and 22)

If we apply those weights* on observations with $x$ between 18 and 22 (since outside that region the weighted points contribute nothing), and calculate the weighted average, that will be the fitted value at 20. In this case it comes out to -106.67 (marked in blue on the plot).

* (these don't sum to 1, which is why we divide by the sum of the weights)

We can then repeat this kind of calculation every place we want to have an estimate of the smoothed function.

$\endgroup$
  • $\begingroup$ Excellent explanation, thank you very much, that all makes much more sense now. $\endgroup$ – CaptainCodeman Dec 19 '15 at 18:02
  • $\begingroup$ top-end answer. $\endgroup$ – javadba Dec 20 '15 at 0:49
3
$\begingroup$

There's a hint in said Wikipedia article that leads to the correct solution. Under the Derivation heading of the Nadaraya-Watson estimator, it links to kernel density estimate.

When performing a kernel density estimate of univariate data, we have that
$\hat{f}_h(x) = \frac{1}{n}\sum_{i=1}^n K_h (x - x_i) = \frac{1}{nh} \sum_{i=1}^n K\Big(\frac{x-x_i}{h}\Big)$

(from the linked page on KDEs.)

Hopefully you can take it from here.

$\endgroup$
  • $\begingroup$ Aha, so I just pass in $(x-x_i) \over h$ as $u$ in my Kernel function and I'm good to go? $\endgroup$ – CaptainCodeman Dec 19 '15 at 12:59
  • $\begingroup$ @CaptainCodeman $K_h(u)$ is $\frac{1}{h}K(\frac{u}{h})$, though in the case of the Nadarya-Watson estimator the scale factor out the front will cancel out (it arises in both numerator and denominator). $\endgroup$ – Glen_b -Reinstate Monica Dec 19 '15 at 15:59
  • $\begingroup$ @Glen_b: excellent answer! (I am relatively new to StackExchange, and cannot comment on your answer.) I wanted to give a hint, and that's all... I suppose it's a remnant of all my tutoring experience. CaptainCodeman: apologies for giving only a hint! It's a habit I will break. $\endgroup$ – JQVeenstra Dec 20 '15 at 13:16
  • $\begingroup$ @JQVeenstra There's some questions here on CrossValidated where your inclination toward giving hints will be very valuable -- see the self-study tag-wiki. We get more self-study questions every day (and more still which should carry the tag but don't have it yet) $\endgroup$ – Glen_b -Reinstate Monica Dec 20 '15 at 23:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.