# Dimensionality reduction (PCA) after splitting a matrix

Lets say I have 4 variables: A1, A2, B1, B2 and I want to simplify my model. I have reason to suspect that A1 and A2 are measuring a similar thing, likewise with B1 and B2

One way to simplify my model (reduce my dimension count) is to run PCA to condense A1 and A2 down into a single variable/component, and B1 and B2 into a second variable/component. This would leave me with Y = A + B

If my matrix contains all 4 variables, and run PCA to extract only 2 components, theres no guarantee that component 1 corresponds to A, and component 2 corresponds to B

If, however, I create 2 matrices where M1 contains A1 and A2, and M2 contains B1 and B2 and then run PCA on the 2 matrices separately extracting only 1 component each then the components would correspond to A and B respectively

My question: is this a valid approach to reduce the dimensions in my dataset?

I'm leaning towards no because by splitting the matrix into 2 and running PCA separately we have no way of modelling any relationship that might exist between A and B. Is that logic correct?

• I'm curious why you want to do this. – roundsquare Aug 17 '16 at 2:26
• Just at the last question: I think, that logic is correct. They don't share the same vectorspace. But what can be done is to leave all 4 variables in one vector space together, and do some "selective pca" which computes the rotation-angle using the criteria from two variables only (but rotates the whole set of variables) – Gottfried Helms Aug 17 '16 at 9:39
• I should add, that one can of course compute pc-scores for the A-items (based on the A-correlation-matrix) and for the B-items (based on the B-correlation matrix) separately. One can then use that scores for the two first pc's together with the A and B-data in one model (via the 4-item plus 2-pc - correlation-matrix) and find the same results as in the matrix-rotations in my example-answer – Gottfried Helms Aug 17 '16 at 12:00

I give an example how this would look like, when you work with the two subset-PC's (PCs of subset A1,A2, PCs of subset B1,B2) in a joint vectorspace.

I just generate manually a correlation matrix with an example, where the subsets have strong correlation in them, but small correlation between them. For manipulation as "selective pca" I use my homebrewn matrix-tool "MatMate", I think the matrix-language is also nice pseudocode...

0) We begin with the cholesky-decomposition of the correlation-matrix to have a common vector space for all items

[23] lad = cholesky(cor)

:      A1:   1.000   0.000   0.000   0.000
:      A2:   0.981   0.195   0.000   0.000
:      B2:   0.335  -0.726   0.601   0.000
:      B2:   0.396  -0.706   0.573   0.126


1.1a) next we find the principal directions of the subset A; the "1..2" in the rotation-call means: use rows 1..2 only for the computation of the rotation-angles:

[27] lad = rot(lad,"pca",1..2)

:      A1:   0.995  -0.098   0.000   0.000
:      A2:   0.995   0.098   0.000   0.000
:      B2:   0.263  -0.755   0.601   0.000
:      B2:   0.325  -0.741   0.573   0.126


1.1b) to be able to work later with the according principal components we add latent items to the matrix which point into the principal directions

[28] lad= {lad,marksp(lad,{1,2}}

:      A1:   0.995  -0.098   0.000   0.000
:      A2:   0.995   0.098   0.000   0.000
:      B2:   0.263  -0.755   0.601   0.000
:      B2:   0.325  -0.741   0.573   0.126
------------------------------------------
:   A-pc1:   1.000   0.000   0.000   0.000
:   A-pc2:   0.000   1.000   0.000   0.000


1.2a) next we find the principal directions of the subset B:

[32] lad = rot(lad,"pca",3..4)

:      A1:   0.367   0.423  -0.480  -0.675
:      A2:   0.220   0.442  -0.398  -0.773
:      B2:   0.997  -0.072   0.000   0.000
:      B2:   0.997   0.072   0.000   0.000
------------------------------------------
:   A-pc1:   0.295   0.435  -0.441  -0.728
:   A-pc2:  -0.750   0.095   0.420  -0.502


[33] lad= {lad,marksp(lad,{1,2}}

:      A1:   0.367   0.423  -0.480  -0.675
:      A2:   0.220   0.442  -0.398  -0.773
:      B2:   0.997  -0.072   0.000   0.000
:      B2:   0.997   0.072   0.000   0.000
------------------------------------------
:   A-pc1:   0.295   0.435  -0.441  -0.728
:   A-pc2:  -0.750   0.095   0.420  -0.502
------------------------------------------
:   B-pc1:   1.000   0.000    .       .
:   B-pc2:   0.000   1.000    .       .


2.1) Now we look at the principal components "second order": the principal components of the two main directions of each subset (marked with "***" here)

[37] lad = rot(lad,"pca",5´7)

:      A1:   0.846   0.529  -0.043   0.042
:      A2:   0.755   0.653   0.043  -0.042
:      B2:   0.783  -0.619  -0.044  -0.047
:      B2:   0.822  -0.566   0.044   0.047
------------------------------------------
:***A-pc1:   0.805   0.594    .       .   ******
:   A-pc2:  -0.466   0.632   0.443  -0.433
------------------------------------------
:***B-pc1:   0.805  -0.594    .       .   ******
:   B-pc2:   0.270   0.366   0.611   0.648


This is very similar to the idea of canonical correlations: the main-directions of subsets of items are pca'ed (in canonical correlations: regressed on each other).
But as the correlation between them (0.805*0.805 - 0.594*0.594) is small, it might be more interesting to assume they rather model orthogonal (or slightly oblique) directions.
2.2) So we rotate to "quartimax" or "varimax" structure between them:

[40] lad = rot(lad,"varimax",5´7,1..2)

:      A1:   0.224   0.973  -0.043   0.042
:      A2:   0.072   0.996   0.043  -0.042
:      B2:   0.991   0.116  -0.044  -0.047
:      B2:   0.981   0.181   0.044   0.047
------------------------------------------
:***A-pc1:   0.149   0.989    .       .   ******
:   A-pc2:  -0.776   0.117   0.443  -0.433
------------------------------------------
:***B-pc1:   0.989   0.149    .       .   ******
:   B-pc2:  -0.068   0.450   0.611   0.648


This looks more convincing as some true model; and while we have the varimax-directions of the main-directions of the two subsets, also the two subsets show the same clear structure of loadings; (besides the small residual-covariance (columns 3 and 4) which remains when we work in a two-dimensional PC-space).

[update] Just for the record. Using the PAF 2-Factor solution we have itempecific variance; this is shown here as additional columns/dimensions in the joint vectorspace. Since the iemspecific variances in my example data were small we do not see significant change in the results, so I only show here the initial PAF solution for the 4 items and the final solution based on the subset-principal components taken in the common-variance-vectorspace only

The initial PAF(2) solution:
PAF
A1:   0.842   0.534    .       .    ||    0.083    .       .       .
A2:   0.749   0.657    .       .    ||     .      0.087    .       .
B2:   0.786  -0.611    .       .    ||     .       .      0.091    .
B2:   0.825  -0.558    .       .    ||     .       .       .      0.089


The intermediate steps are not displayed, except of the final "varimax" rotated solution (varimax PAF-factor 1 and varimax PAF-factor 2 change position compared to the PC-solution above):

PAF = rot(PAF,"varimax",5´7,1..2)
A1:   0.971   0.225    .       .    ||    0.083    .       .       .
A2:   0.994   0.073    .       .    ||     .      0.087    .       .
B2:   0.116   0.989    .       .    ||     .       .      0.091    .
B2:   0.181   0.979    .       .    ||     .       .       .      0.089
------------------------------------------
***A-pc1:   0.989   0.150    .       .    ||     .       .       .       .
A-pc2:   0.149  -0.985    .       .    ||     .       .       .       .
------------------------------------------
***B-pc1:   0.150   0.989    .       .    ||     .       .       .       .
B-pc2:   0.947  -0.143    .       .    ||     .       .       .       .
------------------------------------------

• I've never really thought about rotations in the context of PCA before. When I think of extracting components I think of using SVD on the correlation matrix to pull out the eigenvectors/values. Now, when you talk about rotations it makes me think of confirmatory factor analysis (which, now that I think about it, might make more sense than PCA for the original problem), is your technique similar to CFA in outcome? Because in CFA we can also specify the expected loadings separately for A and B and then perform a rotation to get alignment – Simon Aug 18 '16 at 0:28
• Well, in PCA as well as in FA we have (basically) an euclidean vector space, where the items are mathematically modelled as vectors from the origin; with angles between them which are determined by their correlations (taken as cosine-values of that angles). So rotation is applicable by both methods. And you can do the same method of producing principal components- or principal factor- scores from the pairs of the variables separately and use them as representative central measure of each subset of items. (... contd...) – Gottfried Helms Aug 18 '16 at 1:19
• (..contd...) In MatMate it is also possible to do the FA-computation - one introduces then (in this case) $4$ more columns to the right containing the item-specific error-loadings ( which can be done after the cholesky-decomposition from the correlation matrix with various error-estimations) and do then the same computations as shown above, only on the (now slightly reduced) loadings in the first four columns resp two columns only. – Gottfried Helms Aug 18 '16 at 1:22
• @Simon - as to the question of rotations: you seem to talk about rotations as a mean to fit an apriori model/pattern; in MatMate I've only a "procrustres"-rotation implemented, but no testing yet, because its main purpose is primarily to visualize the internals of statistical and numbertheoretical matrix-operations - initially for my own selfstudy years ago of the whole "linear model"-world in statistics. So if one has finally a grip of what to do, one uses then surely SPSS or R to do this with the real data and to get fit-indices and significance-testing of SE-models. – Gottfried Helms Aug 18 '16 at 1:33

You seem to have two different questions here.

(1) Is this approach valid?

Yes - you will have two variables for each sample and that will be used to predict your output. Assuming you are correct that A1 and A2 measure similar things and B1 and B2 measure similar things, your resulting model should be good (assuming, of course, you use the right model, there is a relationship between A, B, and Y, etc...).

(2) Will it capture any relationship between A and B?

Your dimensionality reduction procedure will not capture any relationship between A and B. However, this is not necessarily a problem. If $A$ and $B$ are correlated, you might have to deal with problems that come from having correlated features, but this is a different problem, not a result or your procedure.

• Regarding your second point, I'm trying to understand why it isnt necessarily a problem. if A and B are actually related/correlated in some way (or if there is even a slight chance that they might be) then doesnt that mean I shouldnt adopt the approach I mentioned? Otherwise any future model involving the A and B components will be incorrect as it wont model the relationship between the 2. If this is a problem then wouldn't it be a result of using my procedure? – Simon Aug 17 '16 at 5:49
• @Simon even if $A$ and $B$ are related, your model for predicting $Y$ should still be fine. Some models deal with correlations between features better than others - once you run through your procedure, you will need to select an appropriate model. – roundsquare Aug 17 '16 at 14:22