Do we lose the bias after PCA?

Question

I'm starting to learn about PCA, and this question just popped into my head. Warning: It may be trivial, since I"m still just starting to learn - sorry!

---

Let's say we've got a whole bunch of $(x,y,z)$ data points, where $x,y$ are the features and $z$ is the label.

We haven't done PCA yet, and we want to do linear regression.

When we center the data, the linear regression plane will automatically pass through zero (so, if we're trying to predict a label on the $Z$ axis with features on the $X$ and $Y$ axes, once we center the data the intercept of a regression plane with the $Z$ axis would be $(0,0,0)$.

In other words, once we center the data, the regression equation $m_xx+m_yy+b=\hat{z}$ ends up having a $b$ of zero.

But, its entirely possible that before we centered the data, there was an $m_xx+m_yy+b=\hat{z}$ with a non-zero $b$ that was the best fit. In other words, when $x$ and $y$ were $0$, $z$ wasn't.

For PCA, we MUST center the data. Which means we lose the bias.

Let's say we do PCA on this data. and come up with a new feature that captures the most variance in the data - some linear combination of the $X$ and $Y$.

Is there any way to recover this bias?

Or, will for some reason the new feature after PCA that we projected our data points onto have no bias once we run a regression on it?

What happens to $b$?!!!

Answer 1

PCA is performing orthogonal rotation. It is necessary to maximize the proportion of variance explained by the components. This rotation allows the component (not the feature) to be used to interpret the model. Bias term is used to explain the average model.

If we are using least squares to fit estimation parameters to a dataset of components with dimension reduction such as PCA applied, and your model contains a bias term, standardizing the data before PCA first will not get rid of the bias term. Bias is a property of the model not the dataset.

A few clarifications:

• Standardizing the data will center the distribution of the dataset.
• The ability to minimize the sum of squared residuals makes no assumptions about the validity of the model, just that the best linear fit to the data can be parameterized.
• The parameter we call the bias term, $b_0$ describing the average model, is fit to an additional column of 1's in X provided we have specified we wish to fit a bias term

Lets take the canonical Python iris data as an example:

from sklearn import datasets
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression


iris = datasets.load_iris()
X = iris.data
y = iris.target

When using bias in combination with the logistic function, the average model is explaining the average value added to the exponent to compute the predicted class. For a binary target variable this is:

$y = \frac{e^{b_0 + b_1x}}{1 + e^{b_0 + b_1x}}$

for a multi class outputs it is a different functional form but similar intuition can be had.

Fitting a model with a bias term to the X will yield 3 intercepts (bias):

lin = LogisticRegression(multi_class='multinomial', solver='lbfgs')
lin.fit(X, y)
lin.intercept_

array([ 9.82061979, 2.22350096, -12.04412075])

Now we transform our dataset, X, by standardizing, i.e. subPCA, keeping just 3 components:

X_standardized = StandardScaler().fit_transform(iris.data) 
X_reduced = PCA(n_components=3).fit_transform(X_reduced)

lin = LogisticRegression(multi_class='multinomial', solver='lbfgs')
lin.fit(X_reduced, y)
lin.intercept_

array([-0.21886586, 2.06804044, -1.84917458])

As you can see the bias term is still the same shape only now it has been shifted as the components explain different proportions of variance than the features. The mean model has changed as has the scale of the components, but bias is still present.