I keep reading about instances where we center the data (e.g., with regularization or PCA) in order to remove the intercept (as mentioned in this question). I know it's simple, but I'm having a hard time intuitively understanding this. Could someone provide the intuition or a reference I can read?
$\begingroup$
$\endgroup$
1
-
2$\begingroup$ This is a very special case of "controlling for other variables" as explained (in several ways) at stats.stackexchange.com/questions/17336/…. The "variable" being controlled for is the constant (intercept) term. $\endgroup$– whuber ♦Commented Oct 22, 2014 at 14:36
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
15
Can these pictures help?
The first 2 pictures are about regression. Centering the data does not alter the slope of regression line, but it makes intercept equal 0.
The pictures below are about PCA. PCA is a regressional model without intercept$^1$. Thus, principal components inevitably come through the origin. If you forget to center your data, the 1st principal component may pierce the cloud not along the main direction of the cloud, and will be (for statistics purposes) misleading.
$^1$ PCA isn't a regression analysis, of course. It however shares formally same linear equation (linear combination) with linear regression. PCA equation is like linear regression equation without intercept - because PCA is a rotation operation.
-
1$\begingroup$ Thanks! Follow-up question: in the case of regression, if I'm predicting y for an unseen x, that means I have to add the intercept back in after prediction, right? And, the intercept would be equal to $\bar{y} - \bar{X}\beta$? $\endgroup$– AlecCommented Feb 6, 2012 at 15:17
-
24$\begingroup$
PCA is maximizing varianceThis is not generally true. PCA maximizes (by the 1st PC) sum-of-squared deviations from the origin. Only if the data were preliminary centered (centering itself isn't a part of PCA) it turns to be maximizing variance. $\endgroup$– ttnphnsCommented Aug 27, 2012 at 11:42 -
3$\begingroup$ P.S. Note that computation of covariances or correlations implies centering $\endgroup$– ttnphnsCommented Aug 27, 2012 at 11:47
-
1$\begingroup$ > P.S. Note that computation of covariances or correlations implies centering – ttnphns Aug 27 '12 at 11:47 While I agree with your other comments, both covariance and correlation do NOT imply centering. Neither cor nor covar change value when an additive constant is applied to the data. $\endgroup$– TPMCommented Oct 30, 2014 at 16:13
-
3$\begingroup$ @ttnphns: Your second figure (about PCA) has one unfortunate drawback: 1st PC on the centered data is exactly orthogonal to the 1st PC on the un-centered data. As a result, it might make an impression that failing to center will only influence the 1st PC, and all the other principal axes will simply get shifted by one (1st become 2nd, etc.). This is of course not so. If your cloud on the left were rotated e.g. clockwise by 60 degrees (and if you also marked 2nd PCs on both subplots), this would become more obvious. $\endgroup$– amoebaCommented Dec 15, 2014 at 11:09
$\begingroup$
$\endgroup$
At least two references that I can find, an earlier edition of which I have been familiar with for about thirty years, state that there are four basic variants of PCA, using:
- Covariance about the mean - this is the variant of PCA which is most commonly referred to as 'PCA' by e.g. sklearn
- Covariance about the origin - this is the variant which results from using sklearn TruncatedSVD
- Correlation about the mean
- Correlation about the origin
PCA 'about the origin' is performed without mean-centring the data, and for 'correlation about', the correlation matrix is used instead of the covariance matrix.
Which variation you use will obviously affect downstream calculations, particularly forms of factor analysis.
From Applications of Factor Analysis to Spectroscopic Methods (Brockwell, 1992):
Correlation about the mean is the traditional form of pre-processing applied before factor analysis, it maintains the Spatial information contained in the data but looses both the origin and the magnitude of the original information. Correlation about the origin maintains the zero point of the data but still looses the relative size information. Covariance about the mean maintains the relative size information but looses the zero point of the data. Covariance about the origin does not alter the data in any way thus preserving the magnitude and origin information. The different techniques find uses depending upon the characteristics of the data; mass spectroscopy data has both an absolute zero point and a common scale for magnitude.
The use of the four pre-treatments above was studied by Rozett and Petersen using the mass spectra of 22 alkyl benzenes and they concluded that with both R and Q analysis (R analysis has the data with rows composed of the samples and columns of spectra, Q analysis is the opposite) the use of covariance about the origin was the best method of pre-treatment as it preserved both the origin of the factor space at zero and also the relative sizes of the components.
References:
Factor Analysis in Chemistry, Third Edition, Malinowski - 2002.
BSR 2949 - Signature Data Processing Final Report - Volume II: Equations and Flow Diagrams, Crawford and Hanson, NASA, 1970.
