3
$\begingroup$

Background

In Introducing Monte Carlo Methods with R by Robert and Casella, in the discussion on "Sampling Importance Resampling", I'm confused by the following argument.

Suppose $f$ and $g$ are pdfs. Sample $X_1 , X_2, \ldots, X_n \sim g$ and then define $X^* = X_i $ with probability $f(X_i)/(ng(X_i))$ (assume for now these probabilities sum to $1$, and I also remark that I assume the authors intend to say they are defining the conditional distribution $X^*|X_1, \ldots, X_n$ even though not stated explicitly...)

The argument is that

$ \begin{aligned} Pr(X^* \in A)&= \sum_{i=1}^n Pr(X^* \in A \text{ and } X^* = X_i) \\ &= \int_{A}\frac{f(x)}{g(x)}g(x) dx \\ &= \int_A f(x) dx \end{aligned} $

so that creating $X^*$ in this way is a valid way to create draws from $f$.

(They then talk about how the ratios $f(X_i)/(ng(X_i))$ must be normalized, but that's not important for my source of confusion.)

My Question

I'm confused by the step that $Pr(X^* \in A \text{ and } X^* = X_i) = \int_A \frac{f(x)}{n g(x)}g(x) dx = \int_A \frac{f(x)}{n} dx$ because $X_i$ is continuous and $X^*$ is discrete. If they were both discrete, and we momentarily considered $g$ a probability MASS function,

$ \begin{aligned} Pr(X^* \in A \text{ and } X^* = X_i) &= Pr(X_i \in A \text{ and } X^* = X_i) \quad \text{ (The event sets are the same )}\\ &= \sum_{a\in A} Pr(X_i =a \text{ and } X^* = X_i) \\ &= \sum_{a\in A} Pr(X_i = a)P(X^* = X_i \mid X_i = a) \\ &= \sum_{a\in A} g(a)\frac{f(a)}{ng(a)}\\ &= \sum_{a\in A} \frac{f(a)}{n}.\\ \end{aligned} $

Now if I converted the sum to an integral, then my proof would be complete. Thus, on a heuristic level, I know how to complete this argument. The problem is I fundamentally don't understand how this argument translates to a continuous $X_i$. In such a case, there is no notion of $P(X^* = X_i \mid X_i = a)$ because when $X_i$ is continuous, this leads to a division by zero $(P(X_i = a) =0)$.

How do I understand this with continuous $X_i$?

$\endgroup$

1 Answer 1

3
$\begingroup$

$X^*$ is not discrete; it's continuous. The $X_i$'s are continuous, too. It is $X^*|X_{1},\ldots,X_n$ that's discrete. I'll copy and paste your $\TeX$ and and write over the stuff that's incorrect. I basically just switched all your sums to integrals. \begin{aligned} Pr(X^* \in A \text{ and } X^* = X_i) &= Pr(X_i \in A \text{ and } X^* = X_i) \quad \text{ (yes )}\\ &= \int_{a \in A} Pr(X_i =a \text{ and } X^* = X_i)da\quad \text{ ($X_i$ is cts )} \\ &= \int_{a\in A} Pr(X_i = a)P(X^* = X_i \mid X_i = a)da \\ &= \int_{a\in A} g(a)\frac{f(a)}{ng(a)}da\\ &= \int_{a\in A} \frac{f(a)}{n}da.\\ \end{aligned}

Notes:

  1. the second and third lines have (joint and marginal) densities written in very terrible ways. This is on purpose. I'm doing this to make it look like your original work

  2. $P(X^* = X_i \mid X_i = a)$ is a pmf. The notation is fine here.

  3. $Pr(X_i =a \text{ and } X^* = X_i)$ is probably not a joint density. I am thinking of this like I am integrating a surface over a diagonal sliver region. Usually a jointly continuous random vector would yield probability zero if I was integrating over a region like this. It's probably not a smooth surface, though. There is mass on these diagonal slivers.

  4. $Pr(X_i = a)$ is a marginal density. It should probably be written like $f_{X_i}(a)$.

  5. We can sum this thing together $n$ times (over $i$) because the events are disjoint

I myself would be interested in hearing about (3).

$\endgroup$
2
  • $\begingroup$ The core explanation indeed is that $X^*_i|X_1,\ldots,X_n$ is discrete, while marginally $X^*_i$ is continuous. $\endgroup$
    – Xi'an
    Jun 2, 2017 at 20:09
  • $\begingroup$ @Xi'an and what about his 3rd point? $\endgroup$ Jul 30, 2018 at 14:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.