Let $(x_1,y_1),\ldots,(x_n,y_n)$ be samples from some unknown distribution $p(X,Y)$ and $\hat{p}(X,Y)$, $\hat{p}(Y)$ density estimates of the joint and marginal distributions (i.e., for the estimation of $\hat{p}(Y)$ we only use the $y$-part of the samples).

Question: Under which condition is

$\int_x \hat{p}(x,y)\ \mathrm{d}x = \hat{p}(y)$?

The answer obviously depends on the estimator. Is it true only in the limit and for consistent estimators? Is it important whether the estimate is parametric or non-parametric?

Essentially, I want to understand why marginalization could be useful in a continuous and non-parametric setting, if I can just as well learn the marginal distribution from the restriction of the data to the marginal dimensions.

  • $\begingroup$ without details about the estimation method you use, the question is unclear as we can always define $\hat{p}(y)$ by $$\int_\mathcal{X} \hat{p}(x,y)\ \mathrm{d}x = \hat{p}(y)$$ $\endgroup$
    – Xi'an
    Feb 4, 2015 at 6:58
  • $\begingroup$ No, you can't. As I wrote, $\hat{p}(y)$ can only use the $y$-part of the samples. I can't be more specific, because the conditions are part of the question. You can assume kernel density estimation, if you need to. $\endgroup$
    – ASML
    Feb 4, 2015 at 7:04

1 Answer 1


$\newcommand{\R}{\mathbb{R}} \newcommand{\ud}{\mathrm{d}}$Suppose we're using KDE, so that $$ \hat{p}(y) = \frac{1}{n} \sum_{i=1}^n k_y(y, y_i) \\ \hat{p}(x, y) = \frac{1}{n} \sum_{i=1}^n k_{xy}\left( \begin{bmatrix}x \\ y\end{bmatrix}, \begin{bmatrix}x_i \\ y_i\end{bmatrix} \right),$$ where $k_y : \R \times \R \to \R$ and $k_{xy} : \R^2 \times \R^2 \to \R$ are our kernels.

Suppose further that $k_{xy}\left( \begin{bmatrix}x \\ y\end{bmatrix}, \begin{bmatrix}x_i \\ y_i\end{bmatrix} \right) = k_y(y, y_i) \, k_{x \mid y}(x, x_i \mid y, y_i)$ for some function $k_{x \mid y}$ which integrates to 1 for any $y, y_i$. If the kernel function is a probability density (i.e. is everywhere nonnegative), then this should hold, since $k_{x \mid y}$ is itself a marginal density; even if not, a product kernel will still satisfy this.

Then \begin{align*} \int \hat{p}(x, y) \,\ud{x} &= \int \frac{1}{n} \sum_{i=1}^n k_{xy}\left( \begin{bmatrix}x \\ y\end{bmatrix}, \begin{bmatrix}x_i \\ y_i\end{bmatrix} \right) \,\ud{x} \\&= \frac{1}{n} \sum_{i=1}^n \int k_y(y, y_i) k_{x \mid y}(x, x_i \mid y, y_i) \,\ud{x} \\&= \frac{1}{n} \sum_{i=1}^n k_y(y, y_i) \int k_{x \mid y}(x, x_i \mid y, y_i) \,\ud{x} \\&= \frac{1}{n} \sum_{i=1}^n k_y(y, y_i) \\&= \hat{p}(y) .\end{align*}

  • $\begingroup$ Thank you, that helped me a lot. So it's not true in case of a full bandwidth matrix, right? Still, since the choice of kernel is almost irrelevant, I don't see why I would ever want to compute the integral in practice... $\endgroup$
    – ASML
    Feb 4, 2015 at 7:59
  • $\begingroup$ @ASML I think it is true for a full bandwidth matrix; see my edit. If you actually need the marginal, though, numerical integration isn't free.... $\endgroup$
    – Danica
    Feb 4, 2015 at 8:01
  • $\begingroup$ But didn't you just show that I basically never need numerical integration, because I can always learn consistent marginals from data? $\endgroup$
    – ASML
    Feb 4, 2015 at 8:06
  • $\begingroup$ @ASML ...yes, I misread what you were saying. :) $\endgroup$
    – Danica
    Feb 4, 2015 at 8:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.