# Modelling dependency between random variables

Assume we have 2 random variables $(X,Y)$ with same distribution $G(.)$. Suppose we are interested in a joint probability of these r.v.'s without any information about dependence or correlation between these random variables.

Would it help to assume $X$ as a linear (or non-linear) function of $Y$?

And lastly, if we knew $corr(X,Y)$ and $F_X(.),F_Y(.)$ could we derive $F_{X,Y}(.)$?

-

One way to address questions like this is to look at extremely simple distributions: that is, those with supports at a small (finite) number of points.

A binary distribution (supported at two points) won't provide enough flexibility, so let's look at ternary distributions. For simplicity, we construct distributions $G$ supported at $-1$, $0$, and $1$, with zero mean, and suppose they are symmetric. Thus the law for $G$ is determined by a single parameter $p$, for which $\Pr_G(X=-1)=\Pr_G(X=1)=p$ and therefore $\Pr_G(X=0)=1-2p$ and necessarily $0 \le p \le 1/2$. Because the mean is zero, the variance of this distribution is the expected value of $X^2$, computed as $p(-1^2) + (1-2p)0^2 + p(1^2)$ = $2p$.

When $X$ and $Y$ both have $G$ for their distribution, bivariate laws for $(X,Y)$ are given by three-by-three contingency tables indexed by the values $(-1,0,1)$. One such family (albeit not the most general one) is

$$\left( \begin{array}{ccc} p^2+\frac{\alpha p}{2}+2 \gamma p & (1-2 p) p-2 p \gamma & p^2-\frac{p \alpha }{2} \\ (1-2 p) p-2 p \gamma & (1-2 p)^2+2 p \gamma & (1-2 p) p \\ p^2-\frac{p \alpha }{2} & (1-2 p) p & p^2+\frac{\alpha p}{2} \end{array} \right)$$

(That this is a possible family of bivariate distributions with identical marginals $G$ can be seen by calculating the row sums and column sums and noting they are both equal to $(p, 1-2p, p)$, which is the distribution $G$.)

This construction will be meaningful if and only if all entries are valid probabilities: that is, they must lie between $0$ and $1$. This restricts $\alpha$ and $\gamma$ in terms of $p$, but nevertheless there are intervals of valid values of both these parameters for all $0 \lt p \lt 1/2$. Here are some plots in $(p, \gamma)$ coordinates for various values of $\alpha$ shown across the top:

We see that $\gamma$ can range from $-1/4$ up to $1/2$, depending on what value(s) we choose for $p$ and $\alpha$, and that $-1 \le \alpha \le 1$ are all possible.

The correlation of $X$ and $Y$ is computed as the expectation of $XY$ divided by their (common) variance. The following expression omits all the terms that are obviously zero:

$$\text{Cor}(X,Y) = [(p^2 + 2\gamma p + \frac{1}{2} \alpha p)(-1)^2 + (p^2 - \frac{1}{2} \alpha p)(-1)(1) + \\(p^2 - \frac{1}{2} \alpha p)(1)(-1) + (p^2 + \frac{1}{2} \alpha p)(1)^2]/(2p) = \alpha + \gamma.$$

Whence the preceding plots indicate, for each possible value of $p$, what the possible range of correlations $\alpha+\gamma$ might be. Because $\alpha+\gamma$ usually determines neither $\alpha$ nor $\gamma$ uniqeuly, this provides a simple concrete example of how knowledge of the correlation can fail to determine the joint distribution.

It is likely that any follow-on questions of a related nature can also be addressed by analyzing this family.

When $X$ is a linear function of $Y$ and, again, $X$ and $Y$ are assumed to have identical distributions, the possibilities are limited. Let the linear function be $X = \beta_0 + \beta_1 Y$. Then

$$\text{Var}(Y) = \text{Var}(X) = \text{Var}(\beta_0+\beta_1 Y) = \beta_1^2 \text{Var}(Y)$$

implies the slope $\beta_1=\pm 1$ because $X$ and $Y$, having identical distributions, must have the same variance. If $G$ is symmetric this allows both values of $\beta_1$, but if $G$ is not symmetric then $\beta_1=1$; in either case necessarily $\beta_0=0$ (for otherwise $X$ and $Y$ have different ranges, which is not possible). Whence

$$\text{Cor}(X,Y) = E[XY]/\sqrt{\text{Var}(X)\text{Var}(Y)} = \beta_1/\beta_1^2 = \beta_1 = \pm 1,$$

showing that the variables are either perfectly correlated or possibly (in the symmetric case) anticorrelated.

-

There is a theory about bivariate and multivariate distributions with specified marginal distributions. For example there are many bivariate distributions with exponential marginal distributions. One due to Marshall and Olkin is well-known. Copulas are bivariate distributions that are constructed by combining specific marginal distributions. An interesting property about bivariate distributions with given marginals is that they do not admit all possible correlations. The limits on the correlations are referred to as the Frechet limits.

I do not think that specifying the correlation is sufficient to uniquely determine a bivariate distribution with fixed marginals.

Specifying X as a linear function of Y with specified slope and intercept parameters would determine the distribution. If X is a nonlinear function of Y and Y is normally distributed X will generally not be normal. Even in the example that Bill Huber provides the pair is not bivariate normal.

-
A lot of this response is rather generic, Michael, because it does not address (a) the question when applied to the case where $X$ is a linear function of $Y$ and (b) it does not exploit the (strong) assumption that the variables have the same marginal distribution. (Notice this implies the correlation can be as large as $1$; when $G$ is symmetric, the correlation can also be as negative as $-1$.) Do you think you could focus your answer more on the specifics of this question? –  whuber Aug 27 '12 at 13:58
@whuber I think I gave a good answer. A linear relationship was not at the heart of the question. The OP is asking for ways to make the bivariate distribution unique and the linear relationship is just one possibility. But I can amend my answer to address it. –  Michael Chernick Aug 27 '12 at 14:22
Concerning your last paragraph: let $Y$ have distribution $F$, assumed symmetric around $\mu$, select any real $a\gt 0$, and define $X=Y$ if $|Y-\mu|\le a$ and $X=2\mu-Y$ otherwise. Then $X$ and $Y$ have the same distribution even though $X$ is a nonlinear function of $Y$. This construction has a positive bearing on your penultimate paragraph, too: you can use it to demonstrate why, in general, the correlation will not determine the bivariate distribution even when the marginals are identical. –  whuber Aug 27 '12 at 14:34
@whuber Can you explain to me how X=2μ−Y outside [-a,a] makes the distribution of X and Y equal? –  Michael Chernick Aug 27 '12 at 14:53
Notice that the transformation is one-to-one a.e. and has the absolute value of its Jacobian equal to $1$ a.e., QED. Alternatively, work out $\Pr(X\le x)$ for representative values of $x$ in the intervals $(-\infty,\mu-a)$, $[\mu-a,\mu+a]$, and $(\mu+a, \infty)$ in terms of the distribution of $Y$ and exploit the (assumed) symmetry about $\mu$. The correlation between $X$ and $Y$ is a continuous function of $a$ with extrema at $-1$ (for $a\to 0$) and $1$ (for $a\to\infty$). Similar but more complicated constructions will have the same properties, proving your penultimate statement. –  whuber Aug 27 '12 at 14:55