17
$\begingroup$

I want to sample from a univariate density $f_X$ but I only know the relationship:

$$f_X(x) = \int f_{X\vert Y}(x\vert y)f_Y(y) dy.$$

I want to avoid the use of MCMC (directly on the integral representation) and, since $f_{X\vert Y}(x\vert y)$ and $f_Y(y)$ are easy to sample from, I was thinking of using the following sampler:

  1. For $j=1,\dots, N$.
  2. Sample $y_j \sim f_Y$.
  3. Sample $x_j \sim f_{X\vert Y}(\cdot\vert y_j)$.

Then, I will end up with the pairs $(x_1,y_1),...,(x_N,y_N)$, and take only the marginal samples $(x_1,\dots,x_N)$. Is this correct?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
13
$\begingroup$

Yes, this is correct. Basically, you have

$$f_{X,Y}(x,y) = f_{X|Y}(x|y) f_Y(y),$$

and as you said, you can sample from the joint density. Picking up just the $x$s from the samples leads you to a sample from the marginal distribution.

This is because the act of ignoring the $y$ is akin to integrating over it. Lets understand this with an example.

Suppose $X$ = Height of mothers and $Y$ = Height of daughter. The goal is to get a sample from $(X,Y)$ to understand the relation between the heights of daughters and their mothers. (I am making the assumption that there is only one daughter in the family, and restricting the population to all daughters over age 18 to ensure full growth).

You go out and get a representative sample $$(x_1, y_1), \dots, (x_N, y_N). $$

Thus for each mother, you have the height of their daughter. There should be a clear relationship between $X$ and $Y$. Now suppose from your dataset, you ignore all the data on the daughters (drop the $Y$), then what do you have? You have exactly heights of randomly chosen mothers which will be $N$ draws from the marginal of $X$.

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ Thank you for this, this is helpful. Do you know if this sampling strategy can be linked to Gibbs sampling in order to formally justify it? $\endgroup$
    – Rod
    May 17, 2016 at 15:48
  • 1
    $\begingroup$ If you can sample from the joint distribution easily, then ignoring the $y$ to get the marginal for $x$ does not need justification. It is a common thing to do. Perhaps, you can say that $y$ is a Linchpin variable, but since you don't need Gibbs sampling to sample from $y$, there is no need for MCMC here. $\endgroup$
    – Greenparker
    May 17, 2016 at 15:51
  • 2
    $\begingroup$ Greenparker, but is there a formal proof of that assertion, i.e. considering only part of the sample taken from the joint gives a sample from the marginal? $\endgroup$
    – An old man in the sea.
    Apr 9, 2018 at 21:51
  • $\begingroup$ Sampling "X=mothers" by sampling (X,Y) and taking X actually gives you samples of "mothers who have exactly one fully grown daughter", which is not is not the same as "mothers". But even if we change your example to say that you are interested in "X=mothers who have exactly one fully grown daughter", getting at X by sampling (X,Y) skews your sample based on the distribution of Y. p(v)=∑(u in support(U))(p(u,v)))=∑(u in support(U))(p(v|u)*p(u)))=(1/sampleSize(u))*∑(u in sample(U))(p(v|u))), because each value of u appears in the sample with probability p(u) - so need to average the p(v|u) draws $\endgroup$
    – radumanolescu
    Feb 23, 2020 at 17:03
  • $\begingroup$ @Anoldmaninthesea.I guess you are confusing between sampling from a joint density, and marginal/joint density calculation. Consider you have a 2X2 table with X axis with value = {1, 2} and Y axes with value ={ 3, 4}. The cells in the 2X2 table define the joint probabilities. If you sample according to these probabilities, you will end up with X values and Y values. Because of their positions along X axis and Y axis, they are the samples from their corresponding marginal distributions. However, only if you want to calculate marginal density would you sum across X or Y axis. $\endgroup$
    – Fred
    Apr 24, 2021 at 0:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.