24
$\begingroup$

How does the inversion method work?
Say I have a random sample $X_1,X_2,...,X_n$ with density $f(x;\theta)={1\over \theta} x^{(1-\theta)\over \theta}$ over
$0<x<1$ and therefore with cdf $F_X(x)=x^{1/\theta}$ on $(0,1)$. Then by the inversion method I get the distribution of $X$ as $F_X^{-1}(u)=u^\theta$.

So does $u^\theta$ has the distribution of $X$? Is this how the inversion method works?

u<-runif(n)
x<-u^(theta)
$\endgroup$
  • 3
    $\begingroup$ See our threads on the probability integral transform. $\endgroup$ – whuber Nov 30 '15 at 19:07
  • 1
    $\begingroup$ Yes, although it's usually called the "probability integral transform." Try deriving the distribution function of $F^{-1}(U)$ to see why it works. $\endgroup$ – dsaxton Nov 30 '15 at 19:08
24
$\begingroup$

The method is very simple, so I'll describe it in simple words. First, take cumulative distribution function $F_X$ of some distribution that you want to sample from. The function takes as input some value $x$ and tells you what is the probability of obtaining $X \leq x$. So

$$ F_X(x) = \Pr(X \leq x) = p $$

inverse of such function function, $F_X^{-1}$ would take $p$ as input and return $x$. Notice that $p$'s are uniformly distributed -- this could be used for sampling from any $F_X$ if you know $F_X^{-1}$. The method is called the inverse transform sampling. The idea is very simple: it is easy to sample values uniformly from $U(0, 1)$, so if you want to sample from some $F_X$, just take values $u \sim U(0, 1)$ and pass $u$ through $F_X^{-1}$ to obtain $x$'s

$$ F_X^{-1}(u) = x $$

or in R (for normal distribution)

U <- runif(1e6)
X <- qnorm(U)

To visualize it look at CDF below, generally, we think of distributions in terms of looking at $y$-axis for probabilities of values from $x$-axis. With this sampling method we do the opposite and start with "probabilities" and use them to pick the values that are related to them. With discrete distributions you treat $U$ as a line from $0$ to $1$ and assign values based on where does some point $u$ lie on this line (e.g. $0$ if $0 \leq u < 0.5$ or $1$ if $0.5 \leq u \leq 1$ for sampling from $\mathrm{Bernoulli}(0.5)$).

enter image description here

Unfortunately, this is not always possible since not every function has its inverse, e.g. you cannot use this method with bivariate distributions. It also does not have to be the most efficient method in all situations, in many cases better algorithms exist.

You also ask what is the distribution of $F_X^{-1}(u)$. Since $F_X^{-1}$ is an inverse of $F_X$, then $F_X(F_X^{-1}(u)) = u$ and $F_X^{-1}(F_X(x)) = x$, so yes, values obtained using such method have the same distribution as $X$. You can check this by a simple simulation

U <- runif(1e6)
all.equal(pnorm(qnorm(U)), U)
| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Good answer. This method works when you have distributions such as exponential, Cauchy, Geometric, Pareto, Logistic, Extreme Value Weibull, etc. For example you can not find a closed form for a normal distribution. Therefore you will not be able to use this method. You could try other methods such as Rejection Method. $\endgroup$ – Abbas Salimi Mar 9 '17 at 6:08
  • 2
    $\begingroup$ You can use the inverse transform method with the normal distribution. There are numerous implementations out there for the normal inverse CDF. For example, you can write normal inverse CDF using the complementary error function. An example implementation of erfc is here. Don't code erfc yourself; use a library. That no closed form formula exists does not imply that you cannot use high quality numerical approximations. $\endgroup$ – Matthew Gunn Mar 9 '17 at 6:32
2
$\begingroup$

Yes, $U^θ$ has the distribution of $X$.

Two additional points on the intuition behind inverse transform method might be useful

(1) In order to understand what $F^{-1}$ actually means please refer to a graph in Tim's answer to Help me understand the quantile (inverse CDF) function

(2) [Please, simply ignore the following, if it brings more confusion instead of clarity]

Let $X$ be any random variable (r.v.) with continuous and strictly increasing cdf $F$. Then $$F(X) \sim \text{Unif}(0,1)$$
Note on notation: $X$ is a r.v. Therefore, the function of r.v. $X$, $F(X)$ is a r.v. itself.

For example, if you would flip the question, so that you would have access to $X$ and wanted to generate a standard uniform, then $X^{1/\theta} \sim \text{Unif}(0,1)$. Let call this random variable $U$. So $$U = X^{1/\theta}$$ Coming back to your question, you have the opposite task: to generate $X$ out of $U$. So, indeed $$X=U^\theta$$

PS. Alternative names for the method are probability integral transform, inverse transform sampling, the quantile transformation, and, in some sources, "the fundamental theorem of simulation".

| cite | improve this answer | |
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.