I'm starting to learn about PCA, and this question just popped into my head. Warning: It may be trivial, since I"m still just starting to learn - sorry!
Let's say we've got a whole bunch of $(x,y,z)$ data points, where $x,y$ are the features and $z$ is the label.
We haven't done PCA yet, and we want to do linear regression.
When we center the data, the linear regression plane will automatically pass through zero (so, if we're trying to predict a label on the $Z$ axis with features on the $X$ and $Y$ axes, once we center the data the intercept of a regression plane with the $Z$ axis would be $(0,0,0)$.
In other words, once we center the data, the regression equation $m_xx+m_yy+b=\hat{z}$ ends up having a $b$ of zero.
But, its entirely possible that before we centered the data, there was an $m_xx+m_yy+b=\hat{z}$ with a non-zero $b$ that was the best fit. In other words, when $x$ and $y$ were $0$, $z$ wasn't.
For PCA, we MUST center the data. Which means we lose the bias.
Let's say we do PCA on this data. and come up with a new feature that captures the most variance in the data - some linear combination of the $X$ and $Y$.
Is there any way to recover this bias?
Or, will for some reason the new feature after PCA that we projected our data points onto have no bias once we run a regression on it?
What happens to $b$?!!!