Let $(x_1,y_1),\ldots,(x_n,y_n)$ be samples from some unknown distribution $p(X,Y)$ and $\hat{p}(X,Y)$, $\hat{p}(Y)$ density estimates of the joint and marginal distributions (i.e., for the estimation of $\hat{p}(Y)$ we only use the $y$-part of the samples).

Question: Under which condition is

$\int_x \hat{p}(x,y)\ \mathrm{d}x = \hat{p}(y)$?

The answer obviously depends on the estimator. Is it true only in the limit and for consistent estimators? Is it important whether the estimate is parametric or non-parametric?

Essentially, I want to understand why marginalization could be useful in a continuous and non-parametric setting, if I can just as well learn the marginal distribution from the restriction of the data to the marginal dimensions.

  • $\begingroup$ without details about the estimation method you use, the question is unclear as we can always define $\hat{p}(y)$ by $$\int_\mathcal{X} \hat{p}(x,y)\ \mathrm{d}x = \hat{p}(y)$$ $\endgroup$ – Xi'an Feb 4 '15 at 6:58
  • $\begingroup$ No, you can't. As I wrote, $\hat{p}(y)$ can only use the $y$-part of the samples. I can't be more specific, because the conditions are part of the question. You can assume kernel density estimation, if you need to. $\endgroup$ – ASML Feb 4 '15 at 7:04

$\newcommand{\R}{\mathbb{R}} \newcommand{\ud}{\mathrm{d}}$Suppose we're using KDE, so that $$ \hat{p}(y) = \frac{1}{n} \sum_{i=1}^n k_y(y, y_i) \\ \hat{p}(x, y) = \frac{1}{n} \sum_{i=1}^n k_{xy}\left( \begin{bmatrix}x \\ y\end{bmatrix}, \begin{bmatrix}x_i \\ y_i\end{bmatrix} \right),$$ where $k_y : \R \times \R \to \R$ and $k_{xy} : \R^2 \times \R^2 \to \R$ are our kernels.

Suppose further that $k_{xy}\left( \begin{bmatrix}x \\ y\end{bmatrix}, \begin{bmatrix}x_i \\ y_i\end{bmatrix} \right) = k_y(y, y_i) \, k_{x \mid y}(x, x_i \mid y, y_i)$ for some function $k_{x \mid y}$ which integrates to 1 for any $y, y_i$. If the kernel function is a probability density (i.e. is everywhere nonnegative), then this should hold, since $k_{x \mid y}$ is itself a marginal density; even if not, a product kernel will still satisfy this.

Then \begin{align*} \int \hat{p}(x, y) \,\ud{x} &= \int \frac{1}{n} \sum_{i=1}^n k_{xy}\left( \begin{bmatrix}x \\ y\end{bmatrix}, \begin{bmatrix}x_i \\ y_i\end{bmatrix} \right) \,\ud{x} \\&= \frac{1}{n} \sum_{i=1}^n \int k_y(y, y_i) k_{x \mid y}(x, x_i \mid y, y_i) \,\ud{x} \\&= \frac{1}{n} \sum_{i=1}^n k_y(y, y_i) \int k_{x \mid y}(x, x_i \mid y, y_i) \,\ud{x} \\&= \frac{1}{n} \sum_{i=1}^n k_y(y, y_i) \\&= \hat{p}(y) .\end{align*}

  • $\begingroup$ Thank you, that helped me a lot. So it's not true in case of a full bandwidth matrix, right? Still, since the choice of kernel is almost irrelevant, I don't see why I would ever want to compute the integral in practice... $\endgroup$ – ASML Feb 4 '15 at 7:59
  • $\begingroup$ @ASML I think it is true for a full bandwidth matrix; see my edit. If you actually need the marginal, though, numerical integration isn't free.... $\endgroup$ – Danica Feb 4 '15 at 8:01
  • $\begingroup$ But didn't you just show that I basically never need numerical integration, because I can always learn consistent marginals from data? $\endgroup$ – ASML Feb 4 '15 at 8:06
  • $\begingroup$ @ASML ...yes, I misread what you were saying. :) $\endgroup$ – Danica Feb 4 '15 at 8:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.