I'm working on a problem from "The Elements of Statistical Learning" (prob. 6.8):

Suppose that for continuous response $Y$ and predictor $X$, we model the joint density of $X, Y$ using a multivariate Gaussian kernel estimator. Note that the kernel in this case would be the product kernel $\phi_{\lambda}(X) \phi_{\lambda}(Y)$.

(a) Show that the conditional mean $E(Y|X)$ derived from this estimate is a Nadaraya-Watson estimator.

(b) Extend this result to classification by providing a suitable kernel for the estimation of the joint distribution of a continuous $X$ and discrete $Y$.

I know that the Nadaraya-Watson estimator is just the weighted average (equation 2.41 and 6.2 in ESL):

$$\hat f (x_0) = \frac{\sum_{i=0}^N K_{\lambda}(x_0, x_i) y_i}{\sum_{i=0}^N K_{\lambda}(x_0, x_i)}$$

Where $K$ in this case would be the multivariate Gaussian kernel function.

I can think about how to extend this to a classification problem, but am not sure how to approach the first part of this question.

Any pointers would be greatly appreciated!


1 Answer 1


The conditional mean is defined by:

$$E(Y|X)\equiv\int y f(y|x) dy$$

Where $f(Y|X)$ is the conditional density. Using the product rule, you can show:


Substituting this back into the integral you get

$$E(Y|X)\equiv\frac{\int y f(y,x) dy}{f(x)}$$

Which is of the form you seek, if you use the kernel density estimator.

  • $\begingroup$ Could you explain why it is not $ \int \frac{y f(y,x)}{f(x)} dy $ ? How did the denominator of the conditional probability get pulled out of the integral? $\endgroup$ Mar 15, 2011 at 22:24
  • $\begingroup$ Because it has no dependence on $y$. Thus in the integral, it is just a constant. $\endgroup$ Mar 16, 2011 at 0:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.