1
$\begingroup$

I'm starting to learn about PCA, and this question just popped into my head. Warning: It may be trivial, since I"m still just starting to learn - sorry!


Let's say we've got a whole bunch of $(x,y,z)$ data points, where $x,y$ are the features and $z$ is the label.

We haven't done PCA yet, and we want to do linear regression.

When we center the data, the linear regression plane will automatically pass through zero (so, if we're trying to predict a label on the $Z$ axis with features on the $X$ and $Y$ axes, once we center the data the intercept of a regression plane with the $Z$ axis would be $(0,0,0)$.

In other words, once we center the data, the regression equation $m_xx+m_yy+b=\hat{z}$ ends up having a $b$ of zero.

But, its entirely possible that before we centered the data, there was an $m_xx+m_yy+b=\hat{z}$ with a non-zero $b$ that was the best fit. In other words, when $x$ and $y$ were $0$, $z$ wasn't.

For PCA, we MUST center the data. Which means we lose the bias.

Let's say we do PCA on this data. and come up with a new feature that captures the most variance in the data - some linear combination of the $X$ and $Y$.

Is there any way to recover this bias?

Or, will for some reason the new feature after PCA that we projected our data points onto have no bias once we run a regression on it?

What happens to $b$?!!!

$\endgroup$
2
  • 1
    $\begingroup$ Ordinary least squares linear regression passes through the mean of the data. So if you centre first then it will pass through the new origin with no intercept. You can then reverse the centring (just undo every step you took to centre) and that will (usually) reintroduce an intercept term $\endgroup$
    – Henry
    Commented May 31, 2019 at 14:49
  • $\begingroup$ @Henry thanks! But, if we center and then make new features using PCA, how can I "reverse the centering"? How can I uncenter along these new features? .....Well, now that I think about it, logically it would make sense that the intercept wouldn't change, since if these new features are a linear combination of the old features, when the new features are zero the old features were zero as well. So I think I understand what you're saying - we just use the bias from the uncentered linear regression in the linear regression after PCA? $\endgroup$ Commented May 31, 2019 at 18:40

1 Answer 1

1
$\begingroup$

PCA is performing orthogonal rotation. It is necessary to maximize the proportion of variance explained by the components. This rotation allows the component (not the feature) to be used to interpret the model. Bias term is used to explain the average model.

If we are using least squares to fit estimation parameters to a dataset of components with dimension reduction such as PCA applied, and your model contains a bias term, standardizing the data before PCA first will not get rid of the bias term. Bias is a property of the model not the dataset.

A few clarifications:

  • Standardizing the data will center the distribution of the dataset.
  • The ability to minimize the sum of squared residuals makes no assumptions about the validity of the model, just that the best linear fit to the data can be parameterized.
  • The parameter we call the bias term, $b_0$ describing the average model, is fit to an additional column of 1's in X provided we have specified we wish to fit a bias term

Lets take the canonical Python iris data as an example:

from sklearn import datasets
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression


iris = datasets.load_iris()
X = iris.data
y = iris.target

When using bias in combination with the logistic function, the average model is explaining the average value added to the exponent to compute the predicted class. For a binary target variable this is:

$y = \frac{e^{b_0 + b_1x}}{1 + e^{b_0 + b_1x}}$

for a multi class outputs it is a different functional form but similar intuition can be had.

Fitting a model with a bias term to the X will yield 3 intercepts (bias):

lin = LogisticRegression(multi_class='multinomial', solver='lbfgs')
lin.fit(X, y)
lin.intercept_

array([ 9.82061979, 2.22350096, -12.04412075])

Now we transform our dataset, X, by standardizing, i.e. subPCA, keeping just 3 components:

X_standardized = StandardScaler().fit_transform(iris.data) 
X_reduced = PCA(n_components=3).fit_transform(X_reduced)

lin = LogisticRegression(multi_class='multinomial', solver='lbfgs')
lin.fit(X_reduced, y)
lin.intercept_

array([-0.21886586, 2.06804044, -1.84917458])

As you can see the bias term is still the same shape only now it has been shifted as the components explain different proportions of variance than the features. The mean model has changed as has the scale of the components, but bias is still present.

$\endgroup$
3
  • 1
    $\begingroup$ Could you explain how your post about logistic regression responds to a question about how ordinary least squares regression changes when one does or does not include an intercept? $\endgroup$
    – whuber
    Commented May 31, 2019 at 15:28
  • $\begingroup$ Hi, thanks for your answer, +1! It may be because I'm a beginner but I'm really confused. I think I understand what you're saying - you're showing me that if I do logistic regression, there's still a bias term after the non-principal components have been dropped. Your answer may answer my question and I may not know enough to understand it, but I was just asking about linear regression and the bias term there... $\endgroup$ Commented May 31, 2019 at 18:38
  • $\begingroup$ I edited my answer to add to my point with standardization (centering) and to hopefully clear up a bit of the confusion. @whuber I used logistic regression as it is convenient and a small extension to model the posterior probabilities Pr(k | x) for classes k. Whether one does or does not include an intercept term to fit the model just depends on whether one wishes to fit a dataset with an average of the model or not. $\endgroup$
    – sjdgg93
    Commented Jun 1, 2019 at 14:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.