2
$\begingroup$

Thanks to @amoeba, I learned that standardized eigenvectors are sometimes calculated, i.e., eigenvectors are divided by the square root of their eigenvalues.

Now, when I want to do principal component regression (PCR), I first calculate the components that I subsequently use in regression. There are two procedures that can be used:

  1. I calculate my components once as raw components, that is I do not standardize the eigenvectors.

  2. I calculate my components as components based on standardized eigenvectors.

Now, I do PCR once with the components in situation 1, and once with the components in situation 2.

My questions:

  • What are the implications (a) for the coefficients, (b) for the interpretation in the two cases?
  • What are the main differences?
  • I know that the variance of the components will differ, but how does that change the behaviour in regression, or does it at all?
  • Is there any recommendation to whether I should use the one but not the other?
Improve this question
$\endgroup$
7
  • 2
    $\begingroup$ maybe you will want to link to where amoeba has dropped the idea you are starting from? Because your question is unclear without that context. $\endgroup$
    – ttnphns
    Commented Apr 10, 2015 at 13:45
  • $\begingroup$ I tried to make it clearer. She didn't really drop the idea, rather uncovered my knowledge lacks. $\endgroup$
    – MaHo
    Commented Apr 10, 2015 at 17:58
  • $\begingroup$ In one case the components are standardized, in the other not. $\endgroup$
    – user73344
    Commented Apr 12, 2015 at 7:48
  • 2
    $\begingroup$ PCA eigenvectors can be multiplied (not divided!) by the square roots of the eigenvalues to obtain loadings. In principal component regression one uses principal components, i.e. projections on the eigenvectors, and not eigenvectors themselves, so I am not sure what exactly you mean by your approach #2. Perhaps you mean standardized principal components (scaled to unit variance). If so, then notice that any scaling of individual predictors does not influence the regression at all, so the answer to your question is that it does not matter. $\endgroup$
    – amoeba
    Commented Jul 8, 2015 at 15:06
  • $\begingroup$ It's common to express eigenvectors as unit vectors. I don't know what dividing each component by its eigenvalue would mean. $\endgroup$
    – duffymo
    Commented Jul 8, 2015 at 16:29

2 Answers 2

Reset to default
0
$\begingroup$

I'm reviewing this and found a nice paper Parameter Estimation in Factor Analysis: Maximum Likelihood versus Principal Component which makes me think that the eigenvectors are indeed standardized, if only for convenient interpretation. They're operating on a standardized nxp data matrix, $\mathbf{Z}$, where the variance matrix has ones on the diagonal and thus there are p units of variance (the trace) "to distribute".

For eigenpairs $(\lambda_k, \mathbf{e_k}), k = 1, .., p$, the sum of the eigenvectors is still $p$ and the proportion of the variance is $\lambda_k / p$. To have the nice interpretation of distributing these p units of variance, you'd clearly have to be working with unit eigenvectors (though the paper doesn't specifically say it).

Improve this answer
$\endgroup$
-1
$\begingroup$

It might have something to do with whether your predictor variables are standardized or not (centered they must be!). If they are in original form, some eigenvector will be extremely large so standardizing them is a good idea (Why?). I don't see the point in standardizing them if your PCA used standardized (commensurating) variables already.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ -1. That's not really an answer to the question. $\endgroup$
    – amoeba
    Commented Apr 14, 2016 at 9:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.