I keep reading about instances where we center the data (e.g., with regularization or PCA) in order to remove the intercept (as mentioned in this question). I know it's simple, but I'm having a hard time intuitively understanding this. Could someone provide the intuition or a reference I can read?
Can these pictures help?
The first 2 pictures are about regression. Centering the data does not alter the slope of regression line, but it makes intercept equal 0.
The pictures below are about PCA. PCA is a regressional model without intercept$^1$. Thus, principal components inevitably come through the origin. If you forget to center your data, the 1st principal component may pierce the cloud not along the main direction of the cloud, and will be (for statistics purposes) misleading.
$^1$ PCA isn't a regression analysis, of course. It however shares formally same linear equation (linear combination) with linear regression. PCA equation is like linear regression equation without intercept - because PCA is a rotation operation.