I keep reading about instances where we center the data (e.g., with regularization or PCA) in order to remove the intercept (as mentioned in this question). I know it's simple, but I'm having a hard time intuitively understanding this. Could someone provide the intuition or a reference I can read?
Can these pictures help?
The first 2 pictures are about regression. Centering the data does not alter the slope of regression line, but it makes intercept equal 0.
The pictures below are about PCA. PCA is a regressional model without intercept$^1$. Thus, principal components inevitably come through the origin. If you forget to center your data, the 1st principal component may pierce the cloud not along the main direction of the cloud, and will be (for statistics purposes) misleading.
$^1$ PCA isn't a regression analysis, of course. It however shares formally same linear equation (linear combination) with linear regression. PCA equation is like linear regression equation without intercept - because PCA is a rotation operation.
At least two references that I can find, an earlier edition of which I have been familiar with for about thirty years, state that there are four basic variants of PCA, using:
- Covariance about the mean - this is the variant of PCA which is most commonly referred to as 'PCA' by e.g. sklearn
- Covariance about the origin - this is the variant which results from using sklearn TruncatedSVD
- Correlation about the mean
- Correlation about the origin
PCA 'about the origin' is performed without mean-centring the data, and for 'correlation about', the correlation matrix is used instead of the covariance matrix.
Which variation you use will obviously affect downstream calculations, particularly forms of factor analysis.
From Applications of Factor Analysis to Spectroscopic Methods (Brockwell, 1992):
Correlation about the mean is the traditional form of pre-processing applied before factor analysis, it maintains the Spatial information contained in the data but looses both the origin and the magnitude of the original information. Correlation about the origin maintains the zero point of the data but still looses the relative size information. Covariance about the mean maintains the relative size information but looses the zero point of the data. Covariance about the origin does not alter the data in any way thus preserving the magnitude and origin information. The different techniques find uses depending upon the characteristics of the data; mass spectroscopy data has both an absolute zero point and a common scale for magnitude.
The use of the four pre-treatments above was studied by Rozett and Petersen using the mass spectra of 22 alkyl benzenes and they concluded that with both R and Q analysis (R analysis has the data with rows composed of the samples and columns of spectra, Q analysis is the opposite) the use of covariance about the origin was the best method of pre-treatment as it preserved both the origin of the factor space at zero and also the relative sizes of the components.
Factor Analysis in Chemistry, Third Edition, Malinowski - 2002.
BSR 2949 - Signature Data Processing Final Report - Volume II: Equations and Flow Diagrams, Crawford and Hanson, NASA, 1970.