# Show that the co-variance between $X_j$ and $X_k$ is $\frac {-pq}{N-1}, j \ne k$

An urn containing $$pN$$ white and $$qN$$ black balls, the total number of balls being $$N$$. Balls are drawn one by one without being returned to the urn until a certain number $$n$$ of balls is reached.

Let $$X_i= \begin{cases} 1&\text{if the i_{th} drawn ball is white }\ \\ 0&\text{if the i_{th} drawn ball is black}\ \end{cases}$$

Show that the co-variance between $$X_j$$ and $$X_k$$ is $$\dfrac {-pq}{N-1}, j \ne k$$

Attempt: The joint probability density function of $$X_j X_k$$ can be computed as :

$$\begin{array}{|c|c|c|c|c|} \hline Y=X_jX_k& (1\times1=1) & (1\times0=0) & (0\times1=0) & (0\times0=0) \\ \hline P(Y)& p^2& pq&pq &q^2\\ \hline \end{array}$$

The co-varirance between $$X_j \text{ and}\ X_k = E [ ~\{X_j - E(X_j) \} \{ X_k-E(X_k)\}~] = E(X_j X_k)-E(X_j)E(X_k)\\ = 1 \times p^2 - [p \times p ] = 0$$

Where could I be going wrong? Is there a conceptual error somewhere?

Thanks a lot for the help

Your calculations assume draws with replacement and therefore yields uncorrelated $$X_j$$ and $$X_k$$. First of all, $$E[X_j]$$ is trivial and equal to $$p$$ as you've also noted. We need to find $$E[X_jX_k]=P(X_j=1,X_k=1)$$.

Here, the key property is $$P(X_j=1,X_k=1)=P(X_1=1,X_2=1)=\frac{pN}{N}\frac{pN-1}{N-1}$$ (to see it, start with $$P(X_1=1,X_3=1)=\sum_{x_2\in\{0,1\}}P(X_1=1,X_2=x_2,X_3=1)$$)

Which yields $$\operatorname{cov}(X_j,X_k)=p\frac{pN-1}{N-1}-p^2=\frac{-pq}{N-1}$$

• Thanks a lot. Could you please explain the last step only. Thanks again Oct 24, 2019 at 20:58
• last step is the covariance (as well as your) formula, i.e. $E[X_jX_k]-E[X_j]E[X_k]$ Oct 24, 2019 at 20:59
• okay. So, if I have understood correctly: The general formula to prove this should be: $P(X_j=1,X_k=1)=\sum_{x \in \{0,1\}} P(X_1=x, \cdots,X_{j-1}=x,X_j=1,X_{j+1}=x,\cdots,X_{k-1}=x, X_k=1),$. Am I correct? Oct 24, 2019 at 21:09
• No, it should be $X_1=x_1,X_2=x_2,...$, not $x$. You'll see that for a given assignment of variables inside the summation, you can permute the variables any way you like w/o disturbing number of $1$'s, which leaves us with $P(X_1=1,X_2=1)$ Oct 24, 2019 at 21:11
• @MathMan yes. And, it'll be quite clear if you start from $X_1,X_3$ case I mentioned above. You'll see that you can permute the numerator terms when number of 1's is constant which will in turn mean that you can actually permute the variables, e.g. $$P(X_1=1,X_2=0,X_3=1)=P(X_1=1,X_2=1,X_3=0)$$ Oct 25, 2019 at 10:40

Visualization:

Suppose that there are $$N=4$$ balls in the urn---one white and three black. We take $$n=3$$ balls from the urn without replacement. We are interested in random variables $$X_1,$$ which takes the value 1 when the first ball is white and 0 otherwise and $$X_3,$$ which takes the value 1 when the third ball is white and 0 otherwise. Intuitively, it is clear that $$X_1$$ and $$X_3$$ are negatively associated because choosing a white ball on the first draw makes it impossible to get a white ball on the third.

Then in @gunes' Answer (+1), we have $$E(X_1) = E(X_3) = p = .25,$$ $$E(X_1X_3) = (.25)(0/4) = 0,$$ and $$Cov(X_1,X_3) = 0 - (.25)^2 = -0.0625.$$

A simulation in R of a million such 3-ball draws gives results that agree with the above to about 3 decimal places.

set.seed(1025)
urn = c(0,0,0,1)
N=10; n=3; m=10^6; x1 = x3 = numeric(m)
for(i in 1:m) {
x = sample(urn, n)
x1[i] = x;  x3[i] = x }

e.1 = mean(x1);  e.3 = mean(x3);  e.13 = mean(x1*x3)
e.1;  e.3;  e.13
 0.249513     # aprx p = 0.25
 0.249795     # aprx p = 0.25
 0

e.13 - e.1*e.3
 -0.0623271   # aprx cov = -0.0625 from above
cov(x1, x3)
 -0.06232716  # aprx cov = -0.0625 using R fcn


A key point is that $$0 = P(X_1 = 1, X_3 = 1) \ne P(X_1 = 1)P(X_3 = 1) = 0.0625.$$

mean(x1==1 & x3==1)
 0
mean(x1==1)* mean(x3==1)
 0.0623271     # aprx 0.0625


One way to show this in a plot is to choose 10,000 of the 1 million points simulated, and jitter them (with random uniform noise) to avoid overplotting. Then the probability at a point is represented by the density of points in a square with the point (red) at its center. The left-hand panel shows results from the simulation above. At right, is a similar plot from balls sampled with replacement, so that draws are independent.