1
$\begingroup$

I am working on propensity score matching using the Kernel Nadaraya–Watson kernel regression. But I am looking to understand the logic of estimation; First, we estimate the Kernel density of each unit (i) in the data using the formula

$$ \hat Y_{0i} = \frac{\sum_{E_0} K\left(\left\| \tfrac{X_i-X_k}{h} \right\|\right) Y_k}{\sum_{E_0} K\left(\left\| \tfrac{X_i-X_k}{h} \right\|\right)} $$

Next, to predict the output for new input X=50, we find the weights that the Gaussian of each data point xi has in the prediction. This can be visualized as the y-axis values corresponding to where the vertical line of x=50 intersects the various Gaussians. This is given below. Finally, we multiply the weight vector w with the label vector y and take its average.

outputs

The formula implies that we estimate the kernel density of a distance between two points, but in practice, we estimate the kernel density of each point (i) in the data. So I am dazed, I am looking to understand this aspect of the formula (I have an ambiguous understanding).

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Let's simplify it. What the model does is it makes a prediction for the labels $\hat y$ by taking a weighted average of the training labels $y_k$,

$$ \hat y = \sum_{k=1}^N w_k y_k $$

with weights such that $w_k \ge 0$ and $\sum_k w_k = 1$. In the case of this model, the weights are calculated using the kernels $K$ applied to the training features $X_k$

$$ w_k = \frac{K\left(\left\| \tfrac{X-X_k}{h} \right\|\right)}{\sum_k K\left(\left\| \tfrac{X-X_k}{h} \right\|\right)} $$

The kernels here serve as measures of familiarity between the test data $X$ and all the training points $X_k$.

TL;DR the prediction is just a weighted average of the training labels weighted in such way that the more similar data points have more weight than the less similar ones, where similarity is measured using the kernels $K$.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thank you for your reply,Yes i understand all of this, my question is relative to the distance, the formula implies that we calculate the distance of our "query point" and the other points in the bandwith h. Because the kernel fonction is a decreasing function, the weight of each unit (i) it depends of the similarity to the query point. So in practice we do not calculate the kernel of a distance, we calculate a kernel density of a point ( if you look a graph you can understand why i do not understand this aspect in the formula). $\endgroup$
    – HABOUDANE Farid
    Sep 2, 2022 at 15:39
  • $\begingroup$ @HABOUDANEFarid but you are calculating the kernel over $X_i - X_k$ (normalized) where $X_i$ is the prediction point and $X_k$ is the training data point, kernel applied to that makes a similarity measure. You compare the $X_i$ point to all the $X_k$ points in training data. $\endgroup$
    – Tim ♦
    Sep 2, 2022 at 15:42
  • $\begingroup$ But some articles apply that the Kernel regression its a local regression. This implies that you compare Xi point to some points Xk points (its depends of the bandwith). Excuse me its difficult for me to understanif u have some documentations help me. $\endgroup$
    – HABOUDANE Farid
    Sep 2, 2022 at 16:16
  • $\begingroup$ @HABOUDANEFarid "bandwidth" is the $h$ parameter, with some kernels it can zero out the weights of the far away points, in this sense some of the points won't be included in the calculations because of having zero weights. But in general, you consider all the data in the training set for making each prediction. Plus, there may be some optimizations so that they can be implemented more efficiently than repeating the calculations every time, but it's a different story. TL;R you calculate the distance of the current point $X$ vs all the $X_k$ points in the training data. $\endgroup$
    – Tim ♦
    Sep 2, 2022 at 17:16
  • $\begingroup$ Ok Mr Tim thank you for your reaction. $\endgroup$
    – HABOUDANE Farid
    Sep 2, 2022 at 17:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.