# How would you decorrelate a collection of vectors so that two vectors are uncorrelated?

Suppose $$X_1, \ldots, X_K$$ are all $$\mathbb{R}^d$$-dimensional random variables each with correlation matrix $$\text{Var}(X_k) = \Sigma_{k} \in \mathbb{R}^{d \times d}$$. Suppose we observe samples $$X_{1k}, \ldots, X_{nk}$$ for each of these samples and that $$X_{ij}$$ and $$X_{ik}$$ could be correlated, in the sense that $$\text{Cov}(X_{ij}, X_{ik}) \neq 0$$ for $$1 \leq j \neq k \leq K$$. We want to decorrelate these vector-valued observations, producing a new set of observations $$\tilde{X}_{1k}, \ldots, \tilde{X}_{nk}$$ so that $$\text{Cov}(\tilde{X}_{ij},\tilde{X}_{ik}) = 0$$ if $$j \neq k$$. I suppose it would be acceptable to even decorrelate the components of the vectors as well so that $$\text{Var}(\tilde{X}_{ik}) = I_d$$ (where $$I_d$$ is the $$d$$-dimensional identity matrix), though I would like to see if it's possible to decorrelate the vectors without also imposing this additional component. How would one do this?

• – jbowman Feb 27 '20 at 20:33
• @jbowman Well PCA is trying to make the covariates in a vector uncorrelated. What about two vectors, making those two vectors uncorrelated? Actually, make it three vectors; all three vectors need to be uncorrelated. – cgmil Feb 28 '20 at 0:03
• AFAICT, you've got $K$ distributions that generate vectors each of length $d$. You draw $n$ samples from each, which would appear to generate $nK$ total vectors, albeit from only $K$ distinct distributions, and you arrange them into $n$ blocks each of size $K$ (that's what I'm deducing, perhaps incorrectly, from working through the indices.) Within each of those $n$ blocks, you want the covariances to be $0$ (except for $j = k$.) It seems to me that PCA, applied to each of the $n$ blocks, would accomplish this, but I may well have misunderstood the structure of the problem. – jbowman Feb 28 '20 at 2:09
• On rereading your comment, it occurs to me that you might be slightly misunderstanding what PCA does. It makes vectors uncorrelated with each other. If I have a $(d \times K)$ matrix with $d \geq K$, it will return a $(d \times K)$ matrix such that each of the $K$ column vectors is uncorrelated with all the others (and a $(K \times K)$ rotation matrix which tells you how to get from the original matrix to the new one.) – jbowman Feb 28 '20 at 2:34