I perform the following experiment: each observer performs 3 tasks to evaluate his strength (S variables), additionally each of them perform 2 IQ tasks. So, I have a dataset with 18 rows for observers and 5 columns: 3 for S and 2 for IQ variables.
I do PCA and for PC1 all 5 loadings are of the same sign, and have similar absolute value (0.4-0.5). For PC2 loadings are in the same absolute value range, but loadings for S and loadings for IQ have different signs.
How can I interpret it?
Can I say that S and IQ are positively correlated (according to PC1)? Or negatively correlated (according to PC2)?
Is it correct interpretation?
Technical details below
1) I perform PCA and study first 3 components for significance. To do it I sample all variables independently with replacement. Thus, I obtain a null-distribution of explained residual Variance for each component (for first component it was obviously explained total variance). I found that observed values for first and second components are very improbable for null-distribution (p-value<<0.01). Thus, I conclude, that 1 and 2 principle components are significant.
2) I check whether I have "noisy" variables in my dataset. To address this question I perform sampling of each variable separately and obtain the null-distribution for case that this variable is independent from others. I found that all variables has similar input in explained residual variance for both first and second components. Perfect.
3) I study the loading of the 1 and 2 principle components. For first all 5 loadings were of the same sign, and has similar absolute value (0.4-0.5). For second principle component loading were in the same absolute value range, but loading for S and loadings for IQ have different sign, and always have same sig inside group. (To simplify S variables loadings always were positive, IQ variables loadings always were negative). To evaluate the stability of this effect i perform sampling of observers with replacement. And obtain null-distribution for each loading for different samples from general populations of observers. They were stable (in less then 1% of samples loadings cross 0).
Thus, I conclude, (A) that first and second component explain significant amount of residual variance, (B) first component loading are of the same size, second component loading dissect 5 variable to 2 groups, which are of a different sign. These groups are the same as biology under variables (physical or mental abilities).
Now comes a most hard part. I should explain these results to my grandmother (boss, student, reviewer... I am not sure what will be easier).
I may say that more muscles you have more clever You are, because all 5 variables projections on first component are cocorrelated (or something like this). But I have no ideas how to explain different sign for "muscles" and "brain" on the second component.
If I will have different signs for a first component loadings everything will be easier.