I'm trying to help someone prepare data for a Linear Mixed Effects model in a community ecology setting. They are using PCA to ensure that the numeric input data is linearly independent. The intended response variable is not being included in the PCA. I think this is a mistake, because it fails to take into consideration how much a dimension of low variance in the explanatory data can impact the response.

Intuitively, if I have two explanatory variables, they fall on a plane. If I run a PCA on those two variables alone (excluding the response variable), and PC1 explains 95% of the variance, I might conclude I can safely ignore PC2. This implies that the response and the explanatory variables together form what is mostly a 2-D space, and linearity implies that the responses lie approximately along a line through that 2-D plane.

Yet we haven't taken the responses into account yet. If in fact the responses vary sharply in the PC2 dimension, then linearity implies they lie roughly not along a 1-D line, but rather lie close to a sharply tilted plane. We will be ignoring that sharp tilt, effectively making predictions that assume the plane of responses is flat with respect to PC2.

My contention is, if we throw out PC2 because PC1 explains 95% of the variance in the explanatory variables alone, then we have a potentially far inferior model to one that leaves PC2 in. My gut tells me that this is a problem, and that it goes away when the response variable is included in the PCA. That is, only then will we be safe in assuming that if PC1 explains 95% of the variance, we are being reasonable in disregarding PC2.

Am I right?

  • $\begingroup$ Is the last "PC1" supposed to be "PC2"? $\endgroup$ – Patrick Malone Apr 8 '18 at 17:00
  • $\begingroup$ That said, it sounds like you're looking for a Partial Least Squares predictor(s) more than a Principal Component. PLS weights the original variables to optimize the relations between the composite and the outcome. $\endgroup$ – Patrick Malone Apr 8 '18 at 17:02
  • $\begingroup$ With respect to "PC1" mis-typed as "PC2" - yes @PatrickMalone, thanks for reading more carefully than I type! $\endgroup$ – nclark Apr 8 '18 at 17:24
  • $\begingroup$ Not sure about Partial Least Squares, still hoping someone can confirm regarding PCA though..... $\endgroup$ – nclark Apr 8 '18 at 17:24

PCA is a data reduction technique. It maximizes the variance in the original variable set accounted for by a relatively small number of components, which are weighted linear composites of the original variables. It does not maximize for predictive utility, but purely for data reduction. Therefore there is no particular reason to expect the component to be well related to the outcome. Whereas there are substantial red flags to incorporating the outcome into the modeling of its predictor.

If you want to maximize for predictive utility of the components, I suggest looking into Partial Least Squares (PLS) modeling (sometimes called PLS-SEM). PLS derives the weights for the PLS variate (component) so as to maximize the variate's correlation with some criterion variable, which can be a single variable or another PLS variate (as one would do with canonical correlation).


The answer to my question became clear after we attempted to write the LME invocation in R. In the rotated coordinate system resulting from the PCA, we no longer have a direct representation of the response variable. There's no way to predict something you can no longer express! This told me my conception of the problem space was completely wrong.

Then I realized that the supposed variance problem I described above is not really a problem. Here's the visualization that made it all come together for me:

Consider the (pre-PCA) 3-D space that includes both explanatory variables and the response variable. The original sample set is a collection of vectors in this space with both explanatory and response variables as components. A linear correlation between explanatory and response variables implies that they fall roughly near a plane cutting through this space.

Now consider the 2-D subspace formed by all positions in our 3-D sample space whose response value is zero. To project the samples onto this 2-D plane, we take just the explanatory components of our sample vectors, replacing all the response coordinates with zeros. This plane and the projected sample values in it is an embedded replica of the 2-D space formed from the explanatory variables alone.

Finally, consider a PCA performed on the explanatory variables alone. If we find that PC1 explains 95% of the variance, we're saying that in the 2-D space with only explanatory variables, all samples are clustered near the PC1 axis. Remembering that this space is identical to the 2-D subspace of projected samples described above, we can infer that the original samples in the full 3-D space are also clustered around PC1, in the explanatory dimensions. We know this because of the shape of the shadow they cast onto the explanatory subspace.

We still don't know for sure that the plane that roughly describes the samples in 3-D is not highly tilted, but since we never stray very far up or down that tilt, we're basically betting on a tilt that is not so extreme that a slight variance in PC2 will throw off the response values very significantly.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.