3
$\begingroup$

I have implemented a framework where two data matrices (with same rows or data points, but different columns or features) are compared by the Pearson correlation of their first principal components (PCs). In this way, I am attempting to measure the similarity of the data sitting on each matrix. This approach so far worked well, but I would like to include few other PCs, while calculating the similarity (or distance).

As a first step, I had concatenated first three PCs to obtain a single vector for each matrix and then compare these vectors with Pearson correlation. However, I am after a more structured way of doing that. Possible solutions might be:

  • After comparing individual PCs, combining this information (e.g. averaging) to get a better estimate.
  • Using a multivariate correlation/similarity measure.

Of course, I am open to other approaches, too! I will very glad if you can steer me into any technique that is mathematically justified.

Improve this question
$\endgroup$
7
  • 1
    $\begingroup$ If you have N PCs and are comparing the first pair across two datasets, what should the similarity be if pc1(dataset1) = pc2(dataset2) and pc1(dataset2) = pc2(dataset1)? Zero? Or something else? Should the amount of variance explained by each PC matter? $\endgroup$
    – bogovicj
    Commented Oct 2, 2020 at 12:06
  • $\begingroup$ Thanks for the response! I only consider element-wise comparisons. First PCs are compared and then seconds, so on.. What I need is a 'single' similarity value after comparing different PCs. Also, I note that I do not delve into variances explained by PCs. $\endgroup$
    – Monotros
    Commented Oct 2, 2020 at 12:11
  • 2
    $\begingroup$ Your approach to concatenate several PCs in a vector before comparing (correlating) such two vectors is among reasonable ones (because PC values are already centered). The averaging approach looks lacking sense because PCs of the same dataset are orthogonal. Multivariate correlation? - what formula is on your mind exactly? $\endgroup$
    – ttnphns
    Commented Oct 2, 2020 at 12:29
  • 1
    $\begingroup$ Please look if my edit of the title suits you. $\endgroup$
    – ttnphns
    Commented Oct 2, 2020 at 12:36
  • 1
    $\begingroup$ Yes, their rows represesent the same axis. $\endgroup$
    – Monotros
    Commented Oct 2, 2020 at 18:08

1 Answer 1

Reset to default
2
$\begingroup$

I take it so that you want to (1) compare the observed datasets, i.e. their data, however (2) you want to compare their PC values rather than the original data values. The observations (cases) are the same in the datasets: row $i$ in one dataset corresponds to row $i$ of the other dataset. Variables (features, columns) could be conceptually comparable as well as not, across the datasets.

Then your idea to vectorize (concatenate) the PCs into a column is obvious and sane. However, it will make difference when you equalize PCs variances before the concatenation or you don't do so.

Concatenation. An example. Dataset V = variables V1 V2 V3. Dataset W = variables W1 W2 W3. Principal component scores of Dataset V= vPC1 vPC2 vPC3. Principal component scores of Dataset W= wPC1 wPC2 wPC3. The PCAs were performed on the covariance matrices, so the PC scores are centered (mean=0) and have variances equal to the components' variances.

  v1       v2       v3       w1       w2       w3     vPC1     vPC2     vPC3     wPC1     wPC2     wPC3

 1.0      1.0      7.0      1.0      3.0      4.0  -2.6987  3.65782   .00800 -3.11221 -1.76367  -.19015
 2.0      2.0      6.0      2.0      2.0      3.0  -1.6755  2.26692  -.12792 -2.03625 -3.12071  -.21763
 3.0      3.0      5.0      6.0      3.0      4.0   -.6523   .87601  -.26385  1.85967 -1.51558   .27774
 2.0      2.0      4.0      2.0      4.0      5.0  -2.1171   .60370 -1.14705 -2.19941  -.30739   .02448
 1.0      3.0      5.0      1.0      5.0      6.0  -2.4920   .88391   .52056 -3.27537  1.04965   .05196
 2.0      4.0      6.0      2.0      4.0      5.0  -1.0272  1.15623  1.40376 -2.19941  -.30739   .02448
 3.0      5.0      5.0      3.0      3.0      4.0   -.0040  -.23468  1.26784 -1.12346 -1.66443  -.00299
 4.0      6.0      4.0      4.0      4.0      5.0   1.0192 -1.62559  1.13191  -.21066  -.20816   .21164
 3.0      5.0      3.0      7.0      5.0      6.0   -.4457 -1.89790   .24871  2.69088  1.34735   .61343
 2.0      4.0      2.0      2.0      6.0      7.0  -1.9105 -2.17021  -.63449 -2.36257  2.50593   .26660
 1.0      3.0      1.0      2.0      5.0      6.0  -3.3754 -2.44253 -1.51769 -2.28099  1.09927   .14554
 2.0      4.0      2.0      2.0      6.0      5.0  -1.9105 -2.17021  -.63449 -2.15537  1.22728 -1.25725
 3.0      5.0      3.0      3.0      7.0      4.0   -.4457 -1.89790   .24871 -1.03536  1.40490 -2.56647
 4.0      6.0      4.0      2.0      5.0      5.0   1.0192 -1.62559  1.13191 -2.17739   .45994  -.61638
 4.0      4.0      5.0      4.0      3.0      6.0    .5917   .31671   .10978  -.33629  -.33617  1.61444
 5.0      5.0      6.0      5.0      4.0      7.0   2.0566   .58903   .99299   .57651  1.12011  1.82907
 6.0      3.0      7.0     15.0      5.0      3.0   2.5490  2.52738  -.42135 10.95669  -.17369  -.92371
 5.0      2.0      7.0      8.0      6.0      5.0   1.3050  3.08668  -.79498  3.81088  1.52498  -.69578
10.0      5.0      5.0      6.0      1.0      2.0   6.4351  -.26234 -1.47762  2.02283 -4.32890   .03563
 7.0      6.0      4.0      7.0      5.0      7.0   3.7788 -1.63744  -.04471  2.58728  1.98668  1.37536

Suppose you are going to compare, by Pearson correlation, (vPC1 & vPC2) with (wPC1 & wPC2). You create two columns: {vPC1;vPC2} and {wPC1;wPC2}. Correlation between them is $.30552$. And is equal to the cosine similarity between them because the PC scores are centered.

Note that this approach put unequal "weight" on the PCs: the variance of the PC1 is greater than the variance of the PC2, so the above correlation value is driven more by similarity between vPC1 and wPC1 than by similarity between vPC2 and wPC2.

If you equalize variances (scales) of all the PCs by z-standardizing them prior the concatenation, PC1 and PC2 will have equal influence on the correlation coefficient which then will come out $.09043$. Again, as the PC scores were centered from the beginning, this is equal to the cosine similarity.

In the current example, similarity (conceptualized as correlation) on PC1 was: r(vPC1,wPC1)=$.61830$ while on PC2 was: r(vPC2,wPC2)=$-.43745$.

Note that $.09043$ above equals the simple average correlation $(.61830+(-.43745))/2$

The sign. The sign of PC scores is arbitrary, you could reverse it. We may reverse the sign in our columns vPC2 and wPC2, for example, without affecting the computed similarity. But what if we decide to reverse the sign only in (say) vPC2 and not wPC2? Do we have the right to do so? In case the initial features (Vs versus Ws) of the two datasets are completely different, then why not? If, for example V2 has nothing to do in advance with W2, why not enter from the beginning W2 as -W2, indeed? Well, I'm saying that there may be situations when it won't be illegal to decide to reverse the sign of a PC in one dataset but not in the other. But under doing this, similarity will change. As we set vPC2 to be -vPC2, r(vPC2,wPC2)=$+.43745$. Then, r({vPC1;vPC2},{wPC1;wPC2}) = $.55626$ without equal weighting and $.52788$ with equal weighting of the PCs 1 and 2. So keep this in mind and watch yourself.

As before, $.52788$ equals the simple average correlation $(.61830+.43745)/2$.

There could be other approaches. You could, for example, average squared correlations (greeting the sign), or average Fisher's z-transformed correlations.

The comparison of two datasets with the same cases but different variables is alias to be the comparison of two sets of variables in the same dataset. The approach exploited by the OP (and illustrated by my answer) is the comparison via extraction, from the two sets, their PCs and correlating between those PCs. If the dominant PCs (the main directions of variablility) correlate, the sets correlate. Another and more direct method to investigate correlations between sets of variables is Canonical Correlation analysis (CCA). Some comparison of PCA and CCA approaches to find here and here.

If your two datasets have different cases (rows) but are comprised of the same variables (features), then to compare their PC structures you have to compare the PCA loadings by means of cosine similarity measure (also called Tucker's coefficient of congruence). Before the comparison, you might want to perform Procrustes rotation, to remove some rotatedness of one loading matrix relative the other. This is a popular method of comparison of factor structures in Factor analysis.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks for the wonderful explanation! I would like to ask that: Have you ever encountered any study/paper that such approaches are used? I am planning to utilize this in my research and it would be great to have some reference. Also, regarding the direction of the PCs, I have followed that: Since my similarity is based on Pearson correlation which is between [-1,1], I can just take the absolute value as PC direction does not matter. $\endgroup$
    – Monotros
    Commented Oct 4, 2020 at 1:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.