3
$\begingroup$

I am trying to perform PCA with varimax rotation on a set of data with 2 variables, in order to orthogonalise them (not for dimension reduction). I used prcomp() to perform PCA and it gave me results like this:

> print(variable.pca)
Standard deviations:
[1] 1.1104314 0.8420727

Rotation:
                  PC1        PC2
LogFreq    -0.6646101 -0.7471903
Simulation -0.7471903  0.6646101

> summary(variable.pca)
Importance of components:
                          PC1    PC2
Standard deviation     1.1104 0.8421
Proportion of Variance 0.6349 0.3651
Cumulative Proportion  0.6349 1.0000

Then I applied varimax() to the variable.pca$rotation. And here's the results:

$loadings

Loadings:
           PC1    PC2   
LogFreq    -0.665 -0.747
Simulation -0.747  0.665

               PC1 PC2
SS loadings    1.0 1.0
Proportion Var 0.5 0.5
Cumulative Var 0.5 1.0

$rotmat
             [,1]          [,2]
[1,] 1.000000e+00 -7.142042e-16
[2,] 7.142042e-16  1.000000e+00

This result is completely uninterpretable to me. I thought that, after varimax rotation, one variable should load heavily on one factor and nearly 0 on the other factor, whereas the other variable should be the reverse. Was there something I did wrong? Or should I not use prcomp() to perform orthogonalisation? I'm pretty new to using PCA for orthogonalisation.

$\endgroup$
5
  • $\begingroup$ You say it is uninterpretable but then in your next sentence you explain what happened. Can you clarify? $\endgroup$
    – mdewey
    Aug 29, 2016 at 14:20
  • $\begingroup$ What I meant by uninterpretable is concerning my aim in doing the PCA. As I said, my aim is for orthogonalisation, that is, to separate my two variables which originally correlated to two factors that do not correlate any more. That's why I used varimax rotation. And my hope is to have the two variables loading on the two new factors separately so that I could argue that one factor is the orthogonalised LogFreq and the other is the orthogonalised Simulation. And I could manage it with SPSS. But the results I've got from R at the moment cannot be interpreted n this way. I hope that makes sense. $\endgroup$
    – Pennie
    Aug 30, 2016 at 12:07
  • 1
    $\begingroup$ I don't know what's going on here. When I use my MatMate-program with that data, the varimax-procedure produces perfectly a two axes solution, where each variable loads on one axis and not on the other one. I'm sure, SPSS would do the same. Perhaps prcomp works only, if there are more than 2 factors/axes? $$ \small \text{loadings = csvdatei("clip") }\\ \small \text{load1 = rot(loadings,"varimax") }\\ \small \text{ load1 = \{\{ 0.00, -1.00 \}, } \\ \small \text{ \{ -1.00, 0.00 \} \} } $$ $\endgroup$ Aug 30, 2016 at 13:03
  • 1
    $\begingroup$ I've done the rotation using SPSS. It actually does the rotation and gives the unit-matrix as result. Here is the list of SPSS (V21)-statements: $$ \small \begin{array} {l} \text{ FACTOR } \\ & \text{ /matrix =in(fac="s:\tstfac1.sav") } \\ & \text{ /ANALYSIS v1 v2 } \\ & \text{ /PRINT EXTRACTION rotation } \\ & \text{ /CRITERIA FACTORS(2) ITERATE(25) } \\ & \text{ /ROTATION varimax } \\ & \text{ /METHOD=CORRELATION. } \\ \end{array} $$ The matrix "tstfac1.sav" contains directly the loadings given by the OP in the "factor-matrix" format. $\endgroup$ Aug 31, 2016 at 21:04
  • 1
    $\begingroup$ @amoeba, I took the OP's 2x2 eigenvectors (Rotation) and varimax (or quartimax) rotated it in SPSS. I suppose Gottfried was doing the same thing. SPSS rotated it to binary-valued matrix, as expected and as it was with a 5x5 eigenvector matrix in the answer stats.stackexchange.com/a/154445/3277. So, SPSS does it correct way. I can't say anything to what's up there with R. $\endgroup$
    – ttnphns
    Sep 1, 2016 at 14:58

1 Answer 1

1
$\begingroup$

General remarks:

  1. One should not varimax-rotate eigenvectors of PCA but loadings of PCA (i.e. eigenvectors scaled up by respective standard deviations).

  2. Also, it does not make sense to rotate all PCA loadings, one would usually choose a subset of leading PCs and rotate those.

If you, nevertheless, take all eigenvectors of PCA and varimax-rotate those (as you did here), you will get the original variables back; i.e. the identity matrix (possibly permuted). See Strange results of varimax rotation of principal component analysis in Stata: rotated components are all zeros and ones. However, it seems that for some strange varimax() in R does not rotate $2\times 2$ matrices correctly.


The non-rotation in the 'VARIMAX' - procedure seems to be a(nother) bug in the R-package.

Checking your result with my matrix-tool MatMate I get the rotation to the identity matrix by the following commands:


a) using my matrix-tool MatMate:

[17] labels = {"LogFreq","Simulation"} '   //' // the [number] is command numbering
                                           // for reference
[18] loadings= {{ -0.6646101, -0.7471903}, _
                { -0.7471903, 0.6646101 }}

 [loadings]      pc1         pc2
   LogFreq: -0.66461    -0.74719
Simulation: -0.74719     0.66461


[20] vm = rot(loadings,"varimax")

  [VARIMAX]      vm1         vm2
   LogFreq:  0.00000    -1.00000
Simulation: -1.00000     0.00000

Now using normalization of the columns to remove artifacts from decimal truncation of the loadings-values (which might have made small deviations from being eigenvectors )

[22] evectors = normsp(loadings)  // columnwise norming to sqsum = 1

[eigvectors]         ev1         ev2
    LogFreq:    -0.66461    -0.74719
 Simulation:    -0.74719     0.66461

// checking the difference of the ev's from the loadings' column-lengthes
[27] chk = (hypothenusesp(loadings)-1)*1e8
[28] disp = {"column_norms(loadings) - 1 = "}||format(chk,6.4)+"e-8"

column_norms(loadings) - 1 =    -3.5282e-8  -3.5282e-8

 // rotate the true eigenvector-type matrix to varimax    
[24] vm = rot(evectors,"varimax")

  [VARIMAX]      vm1         vm2
   LogFreq:  0.00000    -1.00000
Simulation:  -1.00000    0.00000

b) Because there was some concern, that SPSS as well as R might not perform the rotation correctly here also the protocol using SPSS (V21). The following way I got the correct (rotated to unit-matrix) components:

data list records=1 
    /1 ROWTYPE_ 1-8(A) FACTOR_ 10
       LogFreq 12-21 (F,8) Simulation 23-33 (F,8) .
begin data 
FACTOR   1 -0.6646101 -0.7471903 
FACTOR   2 -0.7471903  0.6646101  
END DATA.

FACTOR  
  /MATRIX   IN( FAC = * ) 
  /PRINT    extraction rotation
  /ROTATION varimax    .

Part of the result:

Rotierte Komponentenmatrix ^a
    Komponente  
              1           2
LogFreq      ,000       1,000
Simulation  1,000        ,000

Rotationsmethode: Varimax mit Kaiser-Normalisierung.        
^a Die Rotation ist in 3 Iterationen konvergiert.       
$\endgroup$
3
  • $\begingroup$ +1. I would add that all of that is only of some academic interest; in practice, one should not varimax-rotate eigenvectors of PCA but loadings of PCA (i.e. eigenvectors scaled up by respective standard deviations). Also, it usually does not make sense to rotate all PCA loadings, one would usually choose a subset of leading components and rotate those. Perhaps it would make sense to add something like that to your answer (if you agree), otherwise OP might stay confused about that. $\endgroup$
    – amoeba
    Aug 31, 2016 at 23:24
  • $\begingroup$ @amoeba: very nice, I agree (but I'm lazy at the moment...) Would you mind to insert some nice explanation of this into the post? Otherwise I'll see what I can do in the afternoon... $\endgroup$ Sep 1, 2016 at 6:39
  • $\begingroup$ Sure. I have supplied "general remarks" in the beginning, feel free to edit. $\endgroup$
    – amoeba
    Sep 1, 2016 at 21:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.