Suppose $X_1, \ldots, X_K$ are all $\mathbb{R}^d$-dimensional random variables each with correlation matrix $\text{Var}(X_k) = \Sigma_{k} \in \mathbb{R}^{d \times d}$. Suppose we observe samples $X_{1k}, \ldots, X_{nk}$ for each of these samples and that $X_{ij}$ and $X_{ik}$ could be correlated, in the sense that $\text{Cov}(X_{ij}, X_{ik}) \neq 0$ for $1 \leq j \neq k \leq K$. We want to decorrelate these vector-valued observations, producing a new set of observations $\tilde{X}_{1k}, \ldots, \tilde{X}_{nk}$ so that $\text{Cov}(\tilde{X}_{ij},\tilde{X}_{ik}) = 0$ if $j \neq k$. I suppose it would be acceptable to even decorrelate the components of the vectors as well so that $\text{Var}(\tilde{X}_{ik}) = I_d$ (where $I_d$ is the $d$-dimensional identity matrix), though I would like to see if it's possible to decorrelate the vectors without also imposing this additional component. How would one do this?

  • $\begingroup$ en.wikipedia.org/wiki/Principal_component_analysis ? $\endgroup$ – jbowman Feb 27 '20 at 20:33
  • $\begingroup$ @jbowman Well PCA is trying to make the covariates in a vector uncorrelated. What about two vectors, making those two vectors uncorrelated? Actually, make it three vectors; all three vectors need to be uncorrelated. $\endgroup$ – cgmil Feb 28 '20 at 0:03
  • $\begingroup$ AFAICT, you've got $K$ distributions that generate vectors each of length $d$. You draw $n$ samples from each, which would appear to generate $nK$ total vectors, albeit from only $K$ distinct distributions, and you arrange them into $n$ blocks each of size $K$ (that's what I'm deducing, perhaps incorrectly, from working through the indices.) Within each of those $n$ blocks, you want the covariances to be $0$ (except for $j = k$.) It seems to me that PCA, applied to each of the $n$ blocks, would accomplish this, but I may well have misunderstood the structure of the problem. $\endgroup$ – jbowman Feb 28 '20 at 2:09
  • $\begingroup$ On rereading your comment, it occurs to me that you might be slightly misunderstanding what PCA does. It makes vectors uncorrelated with each other. If I have a $(d \times K)$ matrix with $d \geq K$, it will return a $(d \times K)$ matrix such that each of the $K$ column vectors is uncorrelated with all the others (and a $(K \times K)$ rotation matrix which tells you how to get from the original matrix to the new one.) $\endgroup$ – jbowman Feb 28 '20 at 2:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.