I'm trying to help someone prepare data for a Linear Mixed Effects model in a community ecology setting. They are using PCA to ensure that the numeric input data is linearly independent. The intended response variable is not being included in the PCA. I think this is a mistake, because it fails to take into consideration how much a dimension of low variance in the explanatory data can impact the response.
Intuitively, if I have two explanatory variables, they fall on a plane. If I run a PCA on those two variables alone (excluding the response variable), and PC1 explains 95% of the variance, I might conclude I can safely ignore PC2. This implies that the response and the explanatory variables together form what is mostly a 2-D space, and linearity implies that the responses lie approximately along a line through that 2-D plane.
Yet we haven't taken the responses into account yet. If in fact the responses vary sharply in the PC2 dimension, then linearity implies they lie roughly not along a 1-D line, but rather lie close to a sharply tilted plane. We will be ignoring that sharp tilt, effectively making predictions that assume the plane of responses is flat with respect to PC2.
My contention is, if we throw out PC2 because PC1 explains 95% of the variance in the explanatory variables alone, then we have a potentially far inferior model to one that leaves PC2 in. My gut tells me that this is a problem, and that it goes away when the response variable is included in the PCA. That is, only then will we be safe in assuming that if PC1 explains 95% of the variance, we are being reasonable in disregarding PC2.
Am I right?