# Consistency of density estimation under marginalization

Let $(x_1,y_1),\ldots,(x_n,y_n)$ be samples from some unknown distribution $p(X,Y)$ and $\hat{p}(X,Y)$, $\hat{p}(Y)$ density estimates of the joint and marginal distributions (i.e., for the estimation of $\hat{p}(Y)$ we only use the $y$-part of the samples).

Question: Under which condition is

$\int_x \hat{p}(x,y)\ \mathrm{d}x = \hat{p}(y)$?

The answer obviously depends on the estimator. Is it true only in the limit and for consistent estimators? Is it important whether the estimate is parametric or non-parametric?

Essentially, I want to understand why marginalization could be useful in a continuous and non-parametric setting, if I can just as well learn the marginal distribution from the restriction of the data to the marginal dimensions.

• without details about the estimation method you use, the question is unclear as we can always define $\hat{p}(y)$ by $$\int_\mathcal{X} \hat{p}(x,y)\ \mathrm{d}x = \hat{p}(y)$$ – Xi'an Feb 4 '15 at 6:58
• No, you can't. As I wrote, $\hat{p}(y)$ can only use the $y$-part of the samples. I can't be more specific, because the conditions are part of the question. You can assume kernel density estimation, if you need to. – ASML Feb 4 '15 at 7:04

## 1 Answer

$\newcommand{\R}{\mathbb{R}} \newcommand{\ud}{\mathrm{d}}$Suppose we're using KDE, so that $$\hat{p}(y) = \frac{1}{n} \sum_{i=1}^n k_y(y, y_i) \\ \hat{p}(x, y) = \frac{1}{n} \sum_{i=1}^n k_{xy}\left( \begin{bmatrix}x \\ y\end{bmatrix}, \begin{bmatrix}x_i \\ y_i\end{bmatrix} \right),$$ where $k_y : \R \times \R \to \R$ and $k_{xy} : \R^2 \times \R^2 \to \R$ are our kernels.

Suppose further that $k_{xy}\left( \begin{bmatrix}x \\ y\end{bmatrix}, \begin{bmatrix}x_i \\ y_i\end{bmatrix} \right) = k_y(y, y_i) \, k_{x \mid y}(x, x_i \mid y, y_i)$ for some function $k_{x \mid y}$ which integrates to 1 for any $y, y_i$. If the kernel function is a probability density (i.e. is everywhere nonnegative), then this should hold, since $k_{x \mid y}$ is itself a marginal density; even if not, a product kernel will still satisfy this.

Then \begin{align*} \int \hat{p}(x, y) \,\ud{x} &= \int \frac{1}{n} \sum_{i=1}^n k_{xy}\left( \begin{bmatrix}x \\ y\end{bmatrix}, \begin{bmatrix}x_i \\ y_i\end{bmatrix} \right) \,\ud{x} \\&= \frac{1}{n} \sum_{i=1}^n \int k_y(y, y_i) k_{x \mid y}(x, x_i \mid y, y_i) \,\ud{x} \\&= \frac{1}{n} \sum_{i=1}^n k_y(y, y_i) \int k_{x \mid y}(x, x_i \mid y, y_i) \,\ud{x} \\&= \frac{1}{n} \sum_{i=1}^n k_y(y, y_i) \\&= \hat{p}(y) .\end{align*}

• Thank you, that helped me a lot. So it's not true in case of a full bandwidth matrix, right? Still, since the choice of kernel is almost irrelevant, I don't see why I would ever want to compute the integral in practice... – ASML Feb 4 '15 at 7:59
• @ASML I think it is true for a full bandwidth matrix; see my edit. If you actually need the marginal, though, numerical integration isn't free.... – Danica Feb 4 '15 at 8:01
• But didn't you just show that I basically never need numerical integration, because I can always learn consistent marginals from data? – ASML Feb 4 '15 at 8:06
• @ASML ...yes, I misread what you were saying. :) – Danica Feb 4 '15 at 8:14