9
$\begingroup$

Given two multidimensional datasets, $X$ and $Y$, some people perform multivariable analysis by building a surrogate dependent variable using principal component analysis (PCA). That is, run PCA on $Y$ set, take scores along the first component $y'$, and run a multiple regression of those scores on $X$: $y' = \beta X+\epsilon$. (I am basing my question on this article).

It looks like some adulterated form of canonical correlation analysis (CCA) between the two datasets to me. But having no background in this area I can't put my finger on it. So my question is, what could be the pros/cons of the PCA+regression analysis, compared to CCA?

Intuition says that CCA should be more reasonable here, as (I believe) it builds the canonical variates not to blindly maximize explained variance, but already with the final purpose of maximizing correlation with $X$ in mind. Am I right?


Reference: Mei et al., 2010, Principal-component-based multivariate regression for genetic association studies of metabolic syndrome components

$\endgroup$
6
$\begingroup$

This is a good question, but as it appears from it that you know PCA and CCA a deal, so you are able to answer it yourself. And you do:

[CCA] builds the canonical variates not to blindly [wrt the existence of X] maximize explained variance [in Y], but already with the final purpose of maximizing correlation with X in mind.

Absolutely true. The correlation of the 1st Y's PC with X set will almost always be weaker than the correlation of the 1st Y's CV with it. This comes apparent from pictures comparing PCA with CCA actions.

The PCA+regression you conceive of is two-step, initially "unsupervised" ("blind", as you said) strategy, while CCA is one-step, "supervised" strategy. Both are valid - each in own investigatory settings!

1st principal component (PC1) obtained in PCA of set Y is a linear combination of Y variables. 1st canonical variate (CV1) extracted from set Y in CCA of sets Y and X is a linear combination of Y variables, too. But they are different. (Explore the linked pics, also pay attention to the phrase that CCA is closer to - actually a form of - regression than to PCA.)

PC1 represents set Y. It is the linear summary and the "deputy" from set Y, to face the outer-world relationships later (such as in a subsequent regression of PC1 by variables X).

CV1 represents set X within set Y. It is the linear image of X belonging to Y, the "insider" in Y. The Y-X relationship is already there: CCA is a multivariate regression.

Suppose I've got a children sample's results on a school anxiety questionnaire (such as Phillips test) - Y items, and their results on a social adaptation questionnaire - X items. I want to establish the relationship between the two sets. Items of both inside X and inside Y correlate, but they are quite different and I'm not pleased with the idea to bluntly sum up item scores into a single score in either set, so I'm choosing to stay multivariate.

If I do PCA of Y, extracting PC1, and then regress in on X items, what does it mean? It means that I respect the anxiety questionnaire (Y items) as the sovereign (closed) domain of phenomena, which can express oneself. Express by issuing its best weighted sum of items (accounting for the maximal variance) which represents the whole set Y - its general factor/pivot/trend, "mainstream school anxiety complex", the PC1. It is not before that representation is formed that I turn to the next question how it might be related to social adaptation, the question I'll check in the regression.

If I do CCA of Y vs X, extracting the 1st pair of canonical variates - one from each set - having maximal correlation, what does it mean? It means that I suspect the common factor between (behind) both anxiety and adaptation which makes them correlate with each other. However, I have no reason or ground to extract or model that factor by means of PCA or Factor analysis of the combined set "X variables + Y variables" (because, for example, I see anxiety and adaptation as two quite different domains conceptually, or because the two questionnaires have very different scales (units) or differently-shaped distributions that I fear to "merge", or the number of items is very different in them). I'll be content with just the canonical correlation between the sets. Or I might not be supposing any "common factor" behind the sets, and simply think "X effects Y". Since Y is multivariate the effect is multidimensional, and I'm asking for the 1st-order, strongest effect. It is given by the 1st canonical correlation and the prediction variable corresponding to it is the CV1 of set Y. CV1 is fished out of Y, Y is not selbständig producer of it.

$\endgroup$
  • 1
    $\begingroup$ +1. I would maybe add that CCA, as any other regression, is prone to overfitting. So if Y and/or X include a lot of variables, then doing CCA can result in the first component in Y that is almost 100% predicted from X but is actually entirely due to noise. Doing PCAs on X and Y before doing CCA can act as a kind of regularization. Reducing Y to one PC in an extreme form of it. $\endgroup$ – amoeba says Reinstate Monica Aug 26 '16 at 12:47
  • $\begingroup$ @amoeba, thank you for the addition. It touches the inferential side of the story (population, significance, parsimony) which I completely omitted in the answer. I think I get what you are saying, but you are saying it too reserved, for someone. Overfitting, noise - these things should be explicated, and so I maybe suggest you to issue a separate answer to unwrap your comment. $\endgroup$ – ttnphns Aug 26 '16 at 15:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.