Correlation between two vectors sharing some elements

Question

Suppose $\mathbf{x,y}$ are column vectors of length $n$, where $x_i,y_i \sim \mathcal{D}, \forall i \in \{1,2,\ldots ,n\}$. $\mathbf{z}$ shares $m$ elements with $\mathbf{x}$ and $n-m$ elements with $\mathbf{y}$, that is $\mathbf{z}=[\mathbf{x}_1,\mathbf{x}_2,\ldots,\mathbf{x}_m,\mathbf{y}_{m+1},\mathbf{y}_{m+2}, \ldots,\mathbf{y}_{n-m}]^T$ where $0 \le m \le N$. What is the (expected) correlation between (the observations/elements stored inside) $\mathbf{x}$ and $\mathbf{z}$, defined as $\mathrm{Corr}(\mathbf{x}, \mathbf{z}):=\mathrm{E}[\mathbf{x}-\mathrm{E}[\mathbf{x}])(\mathbf{z}-\mathrm{E}[\mathbf{z}])^T]$?

To make sure my formalization has a mistake above, let me specify the original example question in English. It was to provide an intuition for what correlation is. Suppose there are $n$ dice and we throw then once. We keep $m$ dice and re-throw the $n-m$ dice. What's the correlation between the of sum of first dice throw and the sum of the second dice throw? The result is supposed to be $m/n$, showing that correlation can be interpreted intuitively as the proportion of shared dice (to undergrad stats students).

Answer 1

Numerically, I tried the following Octave code for simulation:

N=60;                            % number of dice
m=3;                             % number of dice changed (not kept) when observed 2nd time
T=1e3;                           % number of experiment repeats (to see the average correlation)
x1=randi(6,[N T]);               % first observation; 10 facets per dice
x2=x1;                           % second observation
x2(1:m,:)=randi(10, [m T]);      % change m dice in second observation
[mean(diag(corr(x1, x2))) 1-m/N] % compute the average/expected correlation
mean(corr(sum(x1), sum(x2)))     % correlation between the sums

Both the correlation of the sums and the correlation of the samples seem to be $m/n$, according to the simulation.

My proof

Let's consider the special cases to elaborate. If $m=0$, none of the elements between $\mathbf{x}$ and $\mathbf{z}$ are shared, and the two are independent. Therefore, their covariance is 0. As a result, their correlation is 0. If $m=n$, then $\mathbf{x} = \mathbf{z}$, thus the correlation is 1. I can show by simulation that, independent of $\mathcal{D}$, $$\mathrm{E}[\mathrm{Corr}(\mathbf{x}, \mathbf{z})] = m/n.$$

In general, consider a vector $\mathbf{w}$ that has $m$ elements that are ones, and $n-m$ elements that are zeros. Then we have: $\mathbf{z}=\mathrm{Diag}(\mathbf{w})\mathbf{x} + \mathrm{Diag}(1-\mathbf{w})\mathbf{y}$ where $\mathrm{Diag}(w)$ is a diagonal matrix with $\mathbf{w}$ along the diagonal. We can then simplify the expression for the correlation using properties of the covariance:

\begin{align} \mathrm{Corr}(\mathbf{x}, \mathbf{z}) &= \frac{\mathrm{Cov}(\mathbf{x}, \mathbf{z})}{\sqrt{\mathrm{Cov}(\mathbf{x}, \mathbf{x}) \mathrm{Cov}(\mathbf{z}, \mathbf{z})}} \\ \\ \mathrm{Cov}(\mathbf{x}, \mathbf{z})&=\mathrm{Cov}(\mathbf{x}, \mathrm{Diag}(\mathbf{w})\mathbf{x} + \mathrm{Diag}(1-\mathbf{w})\mathbf{y})\\ &=\mathrm{Cov}(\mathbf{x}, \mathrm{Diag}(\mathbf{w})\mathbf{x}) + \mathrm{Cov}(\mathbf{x}, \mathrm{Diag}(1-\mathbf{w})\mathbf{y})\\ &=\mathrm{Cov}(\mathbf{x}, \mathbf{x})\mathrm{Diag}(\mathbf{w})^T + \mathrm{Cov}(\mathbf{x}, \mathbf{y})\mathrm{Diag}(1-\mathbf{w})^T\\ &=\mathrm{Var}(\mathbf{x})\mathrm{Diag}(\mathbf{w})\\ \\ \mathrm{Cov}(\mathbf{z}, \mathbf{z})&=\mathrm{Cov}(\mathrm{Diag}(\mathbf{w})\mathbf{x} + \mathrm{Diag}(1-\mathbf{w})\mathbf{y}, \mathrm{Diag}(\mathbf{w})\mathbf{x} + \mathrm{Diag}(1-\mathbf{w})\mathbf{y})\\ &=\mathrm{Diag}(\mathbf{w})\mathrm{Cov}(\mathbf{x}, \mathbf{x})\mathrm{Diag}(\mathbf{w})^T \\ &+ \mathrm{Diag}(\mathbf{1-w})\mathrm{Cov}(\mathbf{y}, \mathbf{y})\mathrm{Diag}(\mathbf{1-w})^T \\ &+\mathrm{Diag}(\mathbf{w})\mathrm{Cov}(\mathbf{x}, \mathbf{y})\mathrm{Diag}(\mathbf{1-w})^T \\ &+\mathrm{Diag}(\mathbf{1-w})\mathrm{Cov}(\mathbf{y}, \mathbf{x})\mathrm{Diag}(\mathbf{w})^T \\ &=\mathrm{Diag}(\mathbf{w})\mathrm{Var}(\mathbf{x})\mathrm{Diag}(\mathbf{w})^T \\ &+ \mathrm{Diag}(\mathbf{1-w})\mathrm{Var}(\mathbf{y})\mathrm{Diag}(\mathbf{1-w})^T\\ &=\mathrm{Diag}(\mathbf{w})\mathrm{Var}(\mathbf{x})\mathrm{Diag}(\mathbf{w})^T \\ &+ \mathrm{Diag}(\mathbf{1-w})\mathrm{Var}(\mathbf{x})\mathrm{Diag}(\mathbf{1-w})^T\\ &=(\mathrm{Diag}(\mathbf{w})+ \mathrm{Diag}(\mathbf{1-w}))\mathrm{Var}(\mathbf{x})(\mathrm{Diag}(\mathbf{w})^T+\mathrm{Diag}(\mathbf{1-w})^T) \\ &=\mathbf{I}\mathrm{Var}(\mathbf{x})\mathbf{I}^T=\mathrm{Var}(\mathbf{x})\\ \end{align}

Above, we used the linearity properties of covariance, and the fact that $\mathbf{x}$ and $\mathbf{y}$ are independent (hence their covariance is 0) and identically distributed (hence their variance are the same).

Putting everything back into the definition of correlation, and the variances canceling out, we have: \begin{align} \mathrm{Corr}(\mathbf{x}, \mathbf{z}) &= \frac{\mathrm{Var}(\mathbf{x}) \mathrm{Diag}(\mathbf{w})}{\sqrt{\mathrm{Var}(\mathbf{x}) \mathrm{Var}(\mathbf{x})}} = \mathrm{Diag}(\mathbf{w}).\\ \end{align}

As the simulation above takes the mean of the diagonal elements, the mean of this diagonal matrix is simply: $$\mathrm{E}[\mathrm{Corr}(\mathbf{x}, \mathbf{z})]= \sum_i \mathbf{w}_i/n = m/n.$$

TL;DR

Note that we showed $\mathrm{Var}(\mathbf{x})=\mathrm{Var}(\mathbf{z})$ above through those verbose steps as one of key simplification. Conceptually, for an observer Alice who only observes $\mathbf{x}$ and another observer Bob who only observes $\mathbf{z}$, they couldn't distinguish the samples; all elements of $\mathbf{z}$ ultimately came from the same distribution as $\mathbf{x}$. Thus, they will have the same statistics, such as variance. Since $\mathbf{x}, \mathbf{z}$ contains $n$ i.i.d elements among which $m$ is shared, the sum along the diagonal of the numerator $\mathrm{Cov}(\mathbf{x}, \mathbf{z})$ of the correlation expression should be $m \mathrm{Var}(\mathbf{x})$. Now all $\mathrm{Var}(\mathbf{x})$ cancels out.