5
$\begingroup$

Assume $X$ and $Y$ are from the same distribution $P$ and $\rho = \frac{Cov(X,Y)}{\sqrt{Var(X)Var(Y)}}$ is fixed. For what kind of $P$ can we determine uniquely the joint distribution of $X,Y$?

I know two distributions with the above property, namely

  1. $P = N(\mu,\sigma^2)$, i.e. normal distribution
  2. $P = Ber(p)$, i.e. Bernoulli distribution

Are there other distributions in this list? Or can we characterize the distributions with this property?

Update Sorry for the confusion. I will explain more about what is the meaning of uniqueness in my mind.

For Bernoulli case, I think there is no ambiguity. If $X,Y \sim Ber(p)$, and their correlation coefficient is $\rho$, then we must have

$$Pr(X=1,y=1) = \mathbb{E} XY = \mathbb{E}X\mathbb{E}Y+Cov(X,Y) = p^2+\rho p(1-p) = p(p+\rho(1-p))$$ $$Pr(X=1,Y=0) = Pr(X=1) - Pr(X=1,Y=1) = p(1-p)(1-\rho)$$

And similarly, $Pr(X=0,Y=1), Pr(X=0,Y=0)$ could be expressed by $p$ and $\rho$.

For Gaussian case, lets assume $X,Y \sim N(0,1)$. I previouly thought that the joint distribution must be Gaussian with mean $0$ and covariance matrix $\begin{bmatrix} 1 &\rho \\ \rho &1 \end{bmatrix}$.

But as the comments and answers pointed out, there could be other joint distributions with the same marignal and correlation. I think in this case, we need to restrict the joint distribution to multivariate Gaussian to guarantee the uniqueness.

So, I think the suitable way to raise this question is to require that the joint distribution is a 'bivariate version' of margianl distribution $P$, just as Ben's answer assumed.

$\endgroup$
11
  • $\begingroup$ Are you assuming the functional form of the joint distribution is known and all we need to determine are the parameter values, or is the functional form not known and we'd like to, if possible, determine it as well as its parameter values? Comparing the two answers posted as of this comment makes the importance of the distinction clear! $\endgroup$
    – jbowman
    Commented May 14 at 4:07
  • $\begingroup$ @jbowman I actually assume that we know $P$, i.e. $X,Y\sim P$ and we know the correlation coefficient. I am wondering under what conditions the joint distribution of $(X,Y)$ is uniquely determined. For example, for the Bernoulli case, if we know $X,Y \sim Ber(p)$ and $\rho$, then we can compute all the probabilities of the form $P(X=i,Y = j)$ hence we can determine the joint distribution. $\endgroup$
    – efsdfmo12
    Commented May 14 at 4:19
  • 2
    $\begingroup$ You can construct all such families. After all, each one is determined by the common distribution of $X$ and $Y$ and one other parameter $\rho;$ so for each value of $\rho,$ all you need to is specify a joint distribution for $(X,Y).$ The trick is to do this in a natural sense where the joint distribution varies continuously with $\rho:$ this suggests introducing a family of copulas to do the work -- and shows just how general this construction is. Notice this implies the answer to your question "for what kind of $P$" is all of them. $\endgroup$
    – whuber
    Commented May 14 at 14:23
  • $\begingroup$ The question is a bit confusing. "the joint distribution could be determined uniquely". What do you mean by the addition of 'unique'? That changes the question entirely, but it is conflicting with the post (you mention the normal distribution as an example). Given two marginal distributions and a correlation, you can always define some joint distribution. E.g. the normal distribution that you mention as a case can give rise to the multivariate normal distribution. However, that is not a unique function. $\endgroup$ Commented May 15 at 6:17
  • $\begingroup$ @SextusEmpiricus Sorry for the confusion. I have updated the post. $\endgroup$
    – efsdfmo12
    Commented May 15 at 19:23

3 Answers 3

9
$\begingroup$

This occurs for distributional families determined by mean and variance

I am assuming here that the joint distribution is assumed to be within some stipulated distributional family (e.g., the normal distribution). The other answer here appears not to make this assumption so our conclusions are different.

While it is not the only case where this is possible, if you are working with a distributional family where the joint distribution is fully determined by the first two moments (joint mean and variance) then it is possible to derive the joint distribution from the marginal distributions and the correlations between the random variables. To see this, first note that for a random vector $\mathbf{X} = (X_1,...,X_n)$, the first two joint moments can be written in parameterised form as:

$$\boldsymbol{\mu} \equiv \mathbb{E}(\mathbf{X}) = \begin{bmatrix} \mu_1 \\ \mu_2 \\ \vdots \\ \mu_n \end{bmatrix} \quad \quad \quad \quad \quad \boldsymbol{\Sigma} \equiv \mathbb{V}(\mathbf{X}) = \begin{bmatrix} \sigma_{1}^2 & \rho_{1,2} \sigma_{1} \sigma_{2} & \cdots & \rho_{1,n} \sigma_{1} \sigma_{n} \\ \rho_{2,1} \sigma_{2} \sigma_{1} & \sigma_{2}^2 & \cdots & \rho_{2,n} \sigma_{2} \sigma_{n} \\ \vdots & \vdots & \ddots & \vdots \\ \rho_{n,1} \sigma_{n} \sigma_{1} & \rho_{n,2} \sigma_{n} \sigma_{2} & \cdots & \sigma_{n}^2 \\ \end{bmatrix}.$$

Knowledge of the marginal distribution of each of the random variables allows you to obtain $\mu_1,...,\mu_n$ and $\sigma_1,...,\sigma_n$ and knowledge of the correlations between the random variables is encapsulated by the values $\rho_{i,j}$. Once all these are known you know the first two joint moments, which then gives you the joint distribution (if this is fully determined by these moments).

The examples you have given of the multivariate normal and Bernoulli distributions are both distributional families that are fully determined by their first two (joint) moments. This is true more broadly for elliptical distributions and it is also true of many other "two parameter" distributions (and also most one parameter distributions and all zero parameter distributions). There is not an exact correspondence here, since there are some examples of "two parameter" distributional families where parameters refer to higher order moments involving higher-order interaction than linear correlation. Nevertheless, it is true for most two parameter multivariate distributional families used in practice.

$\endgroup$
4
  • $\begingroup$ Hi, maybe I misunderstand something. But for exponential distribution $Exp(\lambda)$, it is a one parameter distribution which is determined by the first two moments. If we know $X,Y \sim Exp(\lambda)$ and their correlation coefficient is $\rho$, I feel like we cannot determine the joint distribution. Is there something wrong here? $\endgroup$
    – efsdfmo12
    Commented May 14 at 3:45
  • $\begingroup$ @efsdfmo12: Please see the italicised assumption added to my answer. $\endgroup$
    – Ben
    Commented May 14 at 5:25
  • $\begingroup$ @Ben, This sound like that the condition hold for elliptical distributions. It is so? $\endgroup$
    – markowitz
    Commented May 14 at 14:02
  • $\begingroup$ @markowitz: Yes, that would be a special case --- every elliptical distribution is fully determined by the first two moments of the distribution (assuming they exist). I have updated the answer to note this case. Thanks, Ben. $\endgroup$
    – Ben
    Commented May 14 at 21:41
8
$\begingroup$

This claim is false even for the normal distribution.

Let $p(x,y)$ be the pdf for any bivariate normal distribution with $p(x,y)\ge a>0$ on the rectangle where $x$ and $y$ are within four standard deviations of their means. (A bivariate normal distribution meets this condition for any small enough $a$.)

Then we can construct a different pdf with the same marginals and same correlation.

To do this, first draw a chessboard using the lines $x=\mu_x\pm i\sigma_x$ and $y=\mu_y\pm j\sigma_y$ with $i,j\in\{0,1,2,3,4\}$.

Let $S$ be the white squares on white’s side of the board plus the black squares on black’s side of the board.

Let $D$ be the black squares on white’s side of the board plus the white squares on black’s side of the board.

Let $p’$ be another pdf defined by $$p’(x,y)=p(x,y)+ \begin{cases} \phantom{-}a\text{ if }(x,y)\in S\\ -a\text{ if }(x,y)\in D\\ \phantom{-}0\text{ otherwise} \end{cases}\\$$ where $a$ is as above.

Then $p’$ defines a different distribution with the same marginals and same correlation, so $p$ was not determined uniquely by them.

$\endgroup$
2
  • $\begingroup$ Your reply seems to contradict that of Ben. What do you think about this? $\endgroup$
    – markowitz
    Commented May 15 at 11:59
  • 2
    $\begingroup$ @markowitz - Ben's reply assumes the form of the joint distribution is specified, so all we are doing is finding parameters. Matt F.'s reply assumes we don't know the form of the joint distribution, in which case there are multiple possibilities. $\endgroup$
    – jbowman
    Commented May 15 at 19:48
2
$\begingroup$

The Bernoulli distribution is the only one for which this holds

Specifically, we'll show the following:

Let $P$ be a probability distribution supported on at least three points and let $\rho_0$ be the lowest attainable correlation between random variables with distribution $P$. Then, for any $\rho \in (\rho_0, 1)$, there are infinitely many different bivariate distributions $F$ such that, if $(X, Y) \sim F$, then

  1. $X, Y \sim P$, and
  2. $\operatorname{corr}(X, Y) = \rho$.

To see this, let $X_1 = P^{-1}(U), X_2 = P^{-1}(1-U)$, and $X_3$ an independent copy of $X_1$, where $U \sim \operatorname{Uniform}(0, 1)$. Then $X_1$ and $X_2$ are countermonotonic, and so have correlation $\rho_0$ (see discussion here, for example). Also, without loss of generality, assume that $P$ has mean zero and variance one.

Now, let $(I_1, I_2, I_3)$ be a trinomial random variable with probability vector $(p_1, p_2, p_3)$, and define $$ (X, Y) = (X_1, I_1 X_1 + I_2 X_2 + I_3 X_3). $$

Notice that $Y$ is a mixture of random variables each with distribution $P$, and so $Y \sim P$. Moreover, we have that \begin{align*} \operatorname{Cov}(X, Y) &= \operatorname{Cov}(X_1, I_1 X_1) + \operatorname{Cov}(X_1, I_2 X_2) + \operatorname{Cov}(X_1, I_3 X_3) \\ &= p_1 + p_2 \rho_0. \end{align*}

Hence, for any $0 \leq s \leq \frac{1 - \rho}{1 - \rho_0}$, if we take $p$ to be $$ p = (\rho, 0, 1 - \rho) + s(-\rho_0, 1, -(1 - \rho_0)), $$ then $\operatorname{Cov}(X, Y) = \rho$. Thus, $s$ parameterises a family of probability distributions with the same marginal distributions and correlation.

It remains to show that this the distributions parameterised in this way are indeed distinct. To this end, we consider the joint distribution of $(X, Y)$: \begin{align*} F(x, y) &= p_1 \Pr(X_1 \leq x, X_1 \leq y) + p_2 \Pr(P^{-1}(U) \leq x, P^{-1}(1-U) \leq y) \\ &\qquad + p_3 \Pr(X_1 \leq x, X_3 \leq y) \\ &= p_1 (P(x) \wedge P(y)) + p_2(P(x) + P(y) - 1)_+ + p_3 P(x) P(y). \end{align*}

Using the parameterisation of $p$, we write $F = F_0 + s \cdot \delta F$, so that \begin{align*} \delta F(x, y) &= -\rho_0 (P(x) \wedge P(y)) + (P(x) + P(y) - 1)_+ - (1 - \rho_0) P(x)P(y) \\ &= (P(x) + P(y) - 1)_+ - P(x) P(y) - \rho_0[(P(x) \wedge P(y)) - P(x) P(y)]. \end{align*}

Thus, $s$ parameterises an infinite family of distributions unless $\delta F \equiv 0$. But this only happens if, for any $x, y$ such that $0 < P(x), P(y) < 1$, $$ \rho_0 = \frac{(P(x) + P(y) - 1)_+ - P(x) P(y)}{(P(x) \wedge P(y)) - P(x) P(y)} := G(P(x), P(y)). $$

It's fairly straightforward to show that, if $0 < a < b < 1$, then either $G(a, a) \neq G(b, b)$ or $G(a, b) \neq G(a, a)$. But since $P$ is supported on at least three points, $P$ takes at least two distinct values in $(0, 1)$, and so $G(P(x), P(y))$ is not constant. In particular, it cannot be identically equal to $\rho_0$. Thus, $\delta F \not\equiv 0$, and so $s$ indeed parameterises an infinite family of distinct distributions.

This leaves only the cases where $\rho = \rho_0$ or $1$. In those cases, there really is only one possible joint distribution, namely the countermonotonic $(X_1, X_2)$ and comonotonic $(X_1, X_1)$ respectively.

NOTE: a previous version of this answer did not prove that the $(X, Y)$ didn't have the same distribution for all $s$. Thanks to Sextus Empiricus for pointing out this gap and that the claim is false for the Bernoulli case.

$\endgroup$
6
  • 2
    $\begingroup$ In the Bernoulli case it should be possible. It gives rise to a distribution with for points $(0,0), (0,1), (1,0), (1,1)$ and can be parameterized by three variables. The computation of the marginal distributions and the correlation (also 3 values) from those parameters is an injective function, so you can also compute an inverse. (It is however not surjective so not every combination of marginal distribution and correlation may correspond to a joint distribution) $\endgroup$ Commented May 15 at 6:24
  • 1
    $\begingroup$ So say $P$ is Bernoulli with $p<(1-p)=q$ then we mix 3 distributions, the monic, countermonic, and independent $$\begin{bmatrix} 0 & q \\ p& 0\end{bmatrix}, \begin{bmatrix} p&q-p \\ 0&p\end{bmatrix}, \begin{bmatrix} pq & qq \\ pp& qp\end{bmatrix}$$ I hadn't thought that also for the Bernoulli there would have been multiple joint distributions. So my previous comment is not correct. $\endgroup$ Commented May 15 at 6:46
  • 1
    $\begingroup$ "So my previous comment is not correct." Or maybe not. For a Bernoulli distribution I can change $s$, but it doesn't change the joint distribution. We have that the sum of matrices $M_i$ that changes with the parameter $s$ is zero $$-\rho_0 M_1 + M_2 - (1-\rho_0) M_3 = \begin{bmatrix} 0&0\\0&0\end{bmatrix}$$ $\endgroup$ Commented May 15 at 7:19
  • $\begingroup$ @SextusEmpiricus I think for Bernoulli case, there is only one joint distribution. You could determine all the four probabilities by $\rho$ and $p$. $\endgroup$
    – efsdfmo12
    Commented May 15 at 19:20
  • $\begingroup$ Oh, yes, these are good points. I see that in the Bernoulli case, $X_3$ is already a mixture of $X_1$ with probability $p$ and $X_2$ with probability $q$, and indeed there is only one joint distribution in that case. I think that it should be the only such exception though - I'll try to get my ideas straight about that $\endgroup$ Commented May 16 at 0:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.