# Relationship between “Logistic regression + L1 regularization” and PCA

This is the experiment I have done.

My data contain several hundreds of samples but with over 20k features per sample, so I used logistic regression + L1 regularization (LR+L1) to fit a linear classifier with a small number of features.

It works relatively well, and only about 30 features are left. Then I visualized the dataset with the 30 features by projecting it onto a 2-D space with PCA as shown below.

I am aware that PCA doesn't take class labels into consideration at all, but you could see that somehow PCA projection shows the distinguishing capability of the features selected by LR+L1 pretty well. Yet not perfect, as some red points way in the blue cluster are still classified correctly, and vice versa for some blue points.

Then I thought the hyperplane found by LR+L1 is still in the 30-dimensional space, wondering how it would look like in the 2-D space if projected using the same PCA transformation as the data. I sampled hundreds of points from the hyperplane and projected them. In this particular case, the projected hyperplane happens to be a line (i.e. projected decision boundary) as shown in the figure. I tried a few other examples, not necessarily it's always a line, but often it's close to a line (i.e. PCA1 and PCA2 are highly correlated for the decision boundary after PCA transformation).

What puzzles me is why the projected line happens to have nothing to do with separating the two classes in the 2-D space. Should I expect it to do so? My thought was that if the hyperplance separates the two classes in the 30-D space, then after the same linear transformation, it should separate the two classes in the 2-D space, as well, especially when the projected hyperplane is close to a line.

I think I have a decent understanding of how LR+L1 and PCA work, but I still can't wrap my head around this observation. ## 1 Answer

As you remarked, PCA doesn't know about the labels. In other words there is no guarantee that hyperplane that separates the classes is actually aligned with principal components.

See this notebook for even more drastic example of this fact (hyperplane orthogonal to 1st principal component).

Also, did you mean-center and standardize your data?

• Yes, data are mean-centered and standardize. Your example hasn't been standardized yet. – zyxue Aug 31 '17 at 14:34
• Yes, it was a toy example. Although standarization doesn't make much of a difference, as you can check in updated notebook. – Jakub Bartczuk Aug 31 '17 at 15:55
• I followed your example and explored more at here. Given the projected hyperplane is exactly a line in my example (I am still suspicious about it, but suppose that is indeed the case), what does it mean? Does it mean that the hyperplane happen to be orthogonal to the plane formed by the first two principal components? – zyxue Aug 31 '17 at 16:39