This is the experiment I have done.
My data contain several hundreds of samples but with over 20k features per sample, so I used logistic regression + L1 regularization (LR+L1) to fit a linear classifier with a small number of features.
It works relatively well, and only about 30 features are left. Then I visualized the dataset with the 30 features by projecting it onto a 2-D space with PCA as shown below.
I am aware that PCA doesn't take class labels into consideration at all, but you could see that somehow PCA projection shows the distinguishing capability of the features selected by LR+L1 pretty well. Yet not perfect, as some red points way in the blue cluster are still classified correctly, and vice versa for some blue points.
Then I thought the hyperplane found by LR+L1 is still in the 30-dimensional space, wondering how it would look like in the 2-D space if projected using the same PCA transformation as the data. I sampled hundreds of points from the hyperplane and projected them. In this particular case, the projected hyperplane happens to be a line (i.e. projected decision boundary) as shown in the figure. I tried a few other examples, not necessarily it's always a line, but often it's close to a line (i.e. PCA1 and PCA2 are highly correlated for the decision boundary after PCA transformation).
What puzzles me is why the projected line happens to have nothing to do with separating the two classes in the 2-D space. Should I expect it to do so? My thought was that if the hyperplance separates the two classes in the 30-D space, then after the same linear transformation, it should separate the two classes in the 2-D space, as well, especially when the projected hyperplane is close to a line.
I think I have a decent understanding of how LR+L1 and PCA work, but I still can't wrap my head around this observation.
