What are some techniques for sampling two correlated random variables?

Question

What are some techniques for sampling two correlated random variables:

• if their probability distributions are parameterized (e.g., log-normal)

• if they have non-parametric distributions.

The data are two time series for which we can compute non-zero correlation coefficients. We wish to simulate these data in the future, assuming the historical correlation and time series CDF is constant.

For case (2), the 1-D analogue would be to construct the CDF and sample from it. So I guess, I could construct a 2-D CDF and do the same thing. However, I wonder if there is a way to come close by using the individual 1-D CDFs and somehow linking the picks.

Thanks!

Answer 1

I think what you're looking for is a copula. You've got two marginal distributions (specified by either parametric or empirical cdfs) and now you want to specify the dependence between the two. For the bivariate case there are all kinds of choices, but the basic recipe is the same. I'll use a Gaussian copula for ease of interpretation.

To draw from the Gaussian copula with correlation matrix $C$

1. Draw $(Z=(Z_1, Z_2)\sim N(0, C)$

2. Set $U_i = \Phi(Z_i)$ for $i=1, 2$ (with $\Phi$ the standard normal cdf). Now $U_1, U_2\sim U[0,1]$, but they're dependent.

3. Set $Y_i = F_i^{-1}(U_i)$ where $F_i^{-1}$ is the (pseudo) inverse of the marginal cdf for variable $i$. This implies that $Y_i$ follow the desired distribution (this step is just inverse transform sampling).

Voila! Try it for some simple cases, and look at marginal histograms and scatterpolots, it's fun.

No guarantee that this is appropriate for your particular application though (in particular, you might need to replace the Gaussian copula with a t copula) but this should get you started. A good reference on copula modeling is Nelsen (1999), An Introduction to Copulas, but there are some pretty good introductions online too.

Answer 2

Another popular method is "trivariate reduction" which samples $X_1 \sim Y+Z$ and $X_2 \sim W+Z$ so that the correlation is induced by the random variate $Z$. Note that this is also generalizable to more than 2 dimensions-but is more complicated than the 2-d case. You might think you can only get positive correlations but in fact you can also get negative correlations by using $U$ and $(1-U)$ when generating random variates, this will induce a negative correlation on the distributions.

A third popular method is (NORTA) NORmal To Anything; generate correlated normal variates, make them into uniform random variates via evaluating their respective cdfs, then use these "new" uniform random variates as a source of randomness in generating draws from the new distribution.

Besides the copula (a whole class of methods) approach mentioned in another post, you can also sample from the maximal coupling distribution which is similar in spirit to the copula approach. You specify marginal distributions and the sample from the maximal coupling. This is accomplished by 2 accept-reject steps as described by Pierre Jacob here. Presumably this method can be extended to higher dimensions than 2 but might be more complicated to achieve. Note that the maximal coupling will induce a correlation that depends on the values of the parameters of the marginals see this post for a nice example of this in Xi'an's answer to my question.

If you are willing to accept approximate (in most cases) samples then MCMC techniques are also an option to sample from multi-dimensional distributions.

Also, you could use accept-reject methods but it is typically hard to find a dominating density to sample from and evaluate the ratio of that to the desired density.

This is all the additional methods I can think of but there are probably a couple I missed.