How the kernel regression works?

Question

I am working on propensity score matching using the Kernel Nadaraya–Watson kernel regression. But I am looking to understand the logic of estimation; First, we estimate the Kernel density of each unit (i) in the data using the formula

$$ \hat Y_{0i} = \frac{\sum_{E_0} K\left(\left\| \tfrac{X_i-X_k}{h} \right\|\right) Y_k}{\sum_{E_0} K\left(\left\| \tfrac{X_i-X_k}{h} \right\|\right)} $$

Next, to predict the output for new input X=50, we find the weights that the Gaussian of each data point xi has in the prediction. This can be visualized as the y-axis values corresponding to where the vertical line of x=50 intersects the various Gaussians. This is given below. Finally, we multiply the weight vector w with the label vector y and take its average.

The formula implies that we estimate the kernel density of a distance between two points, but in practice, we estimate the kernel density of each point (i) in the data. So I am dazed, I am looking to understand this aspect of the formula (I have an ambiguous understanding).

Answer 1

Let's simplify it. What the model does is it makes a prediction for the labels $\hat y$ by taking a weighted average of the training labels $y_k$,

$$ \hat y = \sum_{k=1}^N w_k y_k $$

with weights such that $w_k \ge 0$ and $\sum_k w_k = 1$. In the case of this model, the weights are calculated using the kernels $K$ applied to the training features $X_k$

$$ w_k = \frac{K\left(\left\| \tfrac{X-X_k}{h} \right\|\right)}{\sum_k K\left(\left\| \tfrac{X-X_k}{h} \right\|\right)} $$

The kernels here serve as measures of familiarity between the test data $X$ and all the training points $X_k$.

TL;DR the prediction is just a weighted average of the training labels weighted in such way that the more similar data points have more weight than the less similar ones, where similarity is measured using the kernels $K$.