Assume that I have three vectors $A, B, C$ containing information about a set of variables. There may be common information shared between these vectors, that is $A, B, C$ are not independent.
What I would like to achieve is extract the information contained in these vectors and form new, completely independent information vectors. The goal here is to separate the information content of say, $A$ from the content of $B$.
I don't have a lot of knowledge on this topic, but based on what I have learned:
The basic approach would be to use an orthogonalization algorithm, such as the Gram-Schmidt process. Correct me if I am wrong, but this would just be to run a linear regression for $B$ on $A$, take the residuals, and then run another regression for $C$ on $A$ and the residuals from the first regression.
The Gram-Schmidt process would produce linearly independent vectors, however, higher order dependence would still be there. In order to mitigate this, one could add higher order terms in the regression for the Gram-Schmidt. However, is there a better way?
I came across independent component analysis (ICA), which is supposed to result in vectors that are as independent as possible. Apparently though, this is not always the case, and when I tried this with data, the result was basically the same as with PCA, leaving strong higher-order dependence. Have I misunderstood ICA in this respect (what it achieves), and can you give suggestions on methods for generating independent information vectors?
