4
$\begingroup$

I have done multiple regression analysis (MLR) of my data and find out $R^2$ and $r$, and then to remove multicollinearity problem I used PCA. This analysis generated PC equal to my variables, I selected only those factors having eigen value greater than 1 after rotation (varimax rotation). Now I'm confused that if I put factor scores of only selected PC as independent variables in MLR I obtained $r$ and $R^2$ less than those obtained by simple MLR but it should be higher. Where I am wrong?

$\endgroup$
2
  • 1
    $\begingroup$ ...but it should be higher Why? And what is simple MLR? and what is $r$ in your context? $\endgroup$
    – ttnphns
    Jun 27, 2014 at 11:19
  • $\begingroup$ @AndyW, why not turn that into an official answer? $\endgroup$ Jun 27, 2014 at 17:05

1 Answer 1

6
$\begingroup$

If you include all components, not just ones that have an eigenvalue above 1, you will have the same $R^2$. The extracted principal components can just be thought of as linear combinations of the original variables, and these transformed variables need not have any obvious relation to the linear regression with the original variables.

Here is an example in SPSS. So lets make some fake data for 5 X variables. Four of these variables are highly correlated, but one is uncorrelated.

*Making some fake data.
SET SEED 5.
INPUT PROGRAM.
LOOP Id = 1 TO 1000.
END CASE.
END LOOP.
END FILE.
END INPUT PROGRAM.

*Make a set of variables that are highly correlated, except for X5.
COMPUTE #latent = RV.NORMAL(0,1).
VECTOR X(5).
COMPUTE #size = 0.9.
LOOP #i = 1 TO 5.
  DO IF #i <= 4.
    COMPUTE X(#i) = #size*#latent + RV.NORMAL(0,SQRT(1-#size**2)).
  ELSE.
    COMPUTE X(#i) = RV.NORMAL(0,1).
  END IF.
END LOOP.
COMPUTE Y = 5 + 0.2*X1 - 0.3*X2 + 0.4*X3 + 0.1*X4 - 0.2*X5 + RV.NORMAL(0,1).
FREQ VAR X1 TO Y /FORMAT = NOTABLE /STATISTICS = MEAN VARIANCE.

So here X1 to X4 do have an underlying latent variable, but they each uniquely affect Y in different ways. So when we regress all of the X's on Y we get close to the population parameters for the estimates - even with many of the X's highly correlated. The $R^2$ here is only 0.168 - but this is the correct specification given my simulation.

REGRESSION VARIABLES = Y,X1,X2,X3,X4,X5
  /DEPENDENT = Y
  /METHOD = ENTER X1 X2 X3 X4 X5.

Now, if we ran the default FACTORS command, we would only extract one component using the eigenvalue above 1 rule. This is because I did not generate X5 to be exactly orthogonal to the other X's. It ends up being very close to 1 though - and if you looked at the Scree plot you would probably keep it. Here I force SPSS to extract all 5 principle components (pet peeve - it makes no sense to talk about rotation when we are extracting principal components!)

FACTOR VARIABLES = X1 X2 X3 X4 X5
  /CRITERIA = FACTORS(5)
  /PLOT EIGEN
  /SAVE REG (5,PComp).

Now if we include all 5 of these orthogonal principal components it will equal the same $R^2$ as the original equation, 0.168, but the coefficients have no relationship to the simulated data. The $R^2$ will never be higher than what was found including the original measurements.

REGRESSION VARIABLES = Y,PComp1,PComp2,PComp3,PComp4,PComp5
  /DEPENDENT = Y
  /METHOD = ENTER PComp1 PComp2 PComp3 PComp4 PComp5.

If we mindlessly use the eigenvalue over 1 criteria, the $R^2$ decreases to 0.08.

REGRESSION VARIABLES = Y,PComp1,PComp2,PComp3,PComp4,PComp5
  /DEPENDENT = Y
  /METHOD = ENTER PComp1.

This, unfortunately, is a very common procedure in the social sciences where it isn't necessarily warranted. When you do this, you are basically making a case for a congeneric measurement model where the underlying latent variable is what affects Y, and you measure the latent variable using the principal component scores. If the original variables can affect Y in unique ways reducing those variables to their principal component scores is inappropriate. Many times people do it mindlessly just because a few correlations are high - which if you looked at the original regression here the standard errors are small enough that shouldn't be any concern at all.

$\endgroup$
6
  • $\begingroup$ I would be interested to read more about the point you make in your last paragraph. Do you happen to know of a paper that discusses this in more detail? I think I understand the basic meaning - if a small principle component is directly proportional to the outcome, then we definitely want to include it in the model - but I'd like to learn more. $\endgroup$
    – Sycorax
    Jun 27, 2014 at 19:05
  • 2
    $\begingroup$ @user777, my answer at stats.stackexchange.com/a/28493 explains and illustrates this phenomenon. (It even uses PCA to construct the sample dataset.) $\endgroup$
    – whuber
    Jun 27, 2014 at 19:15
  • $\begingroup$ @user777 - the point comes up in the SEM literature when discussing formative measurement models. See Three Cs in measurement models: causal indicators, composite indicators, and covariates (Bollen & Bauldry, 2011). Ken Bollen frequently uses the example of socioeconomic disadvantage to illustrate it. Lets say we were predicting whether an individual commits crime based on various measures like poverty, employment, etc. ... $\endgroup$
    – Andy W
    Jun 27, 2014 at 19:23
  • $\begingroup$ These different measures may separately affect an individuals propensity to commit crime - even though combined in toto they define a persons socio-economic status. $\endgroup$
    – Andy W
    Jun 27, 2014 at 19:23
  • $\begingroup$ @Andy, +1. BTW, do you have a pdf of that article to gift it to me? $\endgroup$
    – ttnphns
    Jun 27, 2014 at 21:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.