I keep reading about instances where we center the data (e.g., with regularization or PCA) in order to remove the intercept (as mentioned in this question). I know it's simple, but I'm having a hard time intuitively understanding this. Could someone provide the intuition or a reference I can read?

  • 2
    $\begingroup$ This is a very special case of "controlling for other variables" as explained (in several ways) at stats.stackexchange.com/questions/17336/…. The "variable" being controlled for is the constant (intercept) term. $\endgroup$ – whuber Oct 22 '14 at 14:36

Can these pictures help?

The first 2 pictures are about regression. Centering the data does not alter the slope of regression line, but it makes intercept equal 0.

enter image description here

The pictures below are about PCA. PCA is a regressional model without intercept$^1$. Thus, principal components inevitably come through the origin. If you forget to center your data, the 1st principal component may pierce the cloud not along the main direction of the cloud, and will be (for statistics purposes) misleading.

enter image description here

$^1$ PCA isn't a regression analysis, of course. It however shares formally same linear equation (linear combination) with linear regression. PCA equation is like linear regression equation without intercept - because PCA is a rotation operation.

  • 1
    $\begingroup$ Thanks! Follow-up question: in the case of regression, if I'm predicting y for an unseen x, that means I have to add the intercept back in after prediction, right? And, the intercept would be equal to $\bar{y} - \bar{X}\beta$? $\endgroup$ – Alec Feb 6 '12 at 15:17
  • 20
    $\begingroup$ PCA is maximizing variance This is not generally true. PCA maximizes (by the 1st PC) sum-of-squared deviations from the origin. Only if the data were preliminary centered (centering itself isn't a part of PCA) it turns to be maximizing variance. $\endgroup$ – ttnphns Aug 27 '12 at 11:42
  • 3
    $\begingroup$ P.S. Note that computation of covariances or correlations implies centering $\endgroup$ – ttnphns Aug 27 '12 at 11:47
  • 1
    $\begingroup$ > P.S. Note that computation of covariances or correlations implies centering – ttnphns Aug 27 '12 at 11:47 While I agree with your other comments, both covariance and correlation do NOT imply centering. Neither cor nor covar change value when an additive constant is applied to the data. $\endgroup$ – TPM Oct 30 '14 at 16:13
  • 1
    $\begingroup$ This is backwards. Additive constants indeed don't affect correlations, but that is because they are subtracted out in the calculations, as @ttphns pointed out. That aside, this isn't a new answer, but a comment. We understand that you don't yet have enough reputation to comment, so this will, I trust, be moved by a user with enough reputation after I flag it. $\endgroup$ – Nick Cox Oct 30 '14 at 16:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.