0
$\begingroup$

Assume that I have three vectors $A, B, C$ containing information about a set of variables. There may be common information shared between these vectors, that is $A, B, C$ are not independent.

What I would like to achieve is extract the information contained in these vectors and form new, completely independent information vectors. The goal here is to separate the information content of say, $A$ from the content of $B$.

I don't have a lot of knowledge on this topic, but based on what I have learned:

The basic approach would be to use an orthogonalization algorithm, such as the Gram-Schmidt process. Correct me if I am wrong, but this would just be to run a linear regression for $B$ on $A$, take the residuals, and then run another regression for $C$ on $A$ and the residuals from the first regression.

The Gram-Schmidt process would produce linearly independent vectors, however, higher order dependence would still be there. In order to mitigate this, one could add higher order terms in the regression for the Gram-Schmidt. However, is there a better way?

I came across independent component analysis (ICA), which is supposed to result in vectors that are as independent as possible. Apparently though, this is not always the case, and when I tried this with data, the result was basically the same as with PCA, leaving strong higher-order dependence. Have I misunderstood ICA in this respect (what it achieves), and can you give suggestions on methods for generating independent information vectors?

Improve this question
$\endgroup$
11
  • $\begingroup$ Could you explain how one could even recognize "independence" among vectors, much less somehow eliminate it through linear combinations or any other transformation? $\endgroup$ – whuber♦ Feb 4 '19 at 14:26
  • $\begingroup$ As I wrote, I don't have a lot of knowledge on this topic. However, and please correct me if I am wrong because I want to learn more about this, I am under the impression that one can judge independence based on the ranks of the values in the vectors. For example, if you plot $rank(A)$ on y axis and $rank(B)$ on x axis, independent vectors would show just noise, whereas dependence, be it linear or higher order, would show up as patterns or clumps in the graph. I suppose one can quantify this approach somehow to formally judge independence. Am I completely wrong here? $\endgroup$ – Andrei1234 Feb 4 '19 at 15:59
  • $\begingroup$ In your comment you appear to be viewing these vectors as if their components were random samples from some bivariate distribution $(A,B)$ and you wish to test the hypothesis that $A$ and $B$ are independent. That can be done. (A test can be based on the Spearman correlation coefficient, for instance.) However, your post asks to "form new, completely independent information vectors." That needs to be clarified: what rules of transformation are permitted and what assumptions will you make about the joint distribution of $(A,B)$? $\endgroup$ – whuber♦ Feb 4 '19 at 16:50
  • $\begingroup$ Ah ok, it seems that I messed up with terminology in the question. Yes, indeed, I view the vectors as if their components are samples from some bivariate distribution. That being said, I would like to find a way to form new vectors that are independent (or as independent as possible) of each other. PCA achieves this in a linear sence, correct? Gram-Schmidt can be made to include higher order terms, however, is there a more sophisticated way? $\endgroup$ – Andrei1234 Feb 4 '19 at 16:56
  • $\begingroup$ Yes--but how do you want to "form new vectors" and what will you assume about the bivariate distribution? $\endgroup$ – whuber♦ Feb 4 '19 at 16:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.