After using PCA to reduce dimensionality, does PCA preserve linear separability for any linearly separable set? Will the data still be linearly separable after the transformation? I am thinking that it does preserve linear separability because the PCA just reduces the dimension and not the relationship between the points in terms of separability. Am I on the right line of thinking?
No, it may be that the discriminative information is in the direction of a principal component that explains a relatively small amount of the total variance, and hence gets discarded.
Consider a two-dimensional dataset where the two classes lie in long parallel cigar-shaped elongated Gaussian clusters with a small gap between them. Most of the variance lies along the long axis of the clusters, so the first PC will be in that direction and the second will be orthogonal to it. The data are indistinguishable from the first component, so if you discard the second component, the data will no longer be separable.
No, reducing dimensionality with PCA will only maximize variance, which may or may not translate to linear separability.
Here are two visualizations of variance and separability in opposition. In both cases, the discriminative information lies primarily along the low-variance axis, which would get discarded by rote dimensionality reduction.
Minimal example of three data points (adapted from whuber's comment)
Example of parallel, elongated Gaussian clusters (adapted from Dikran's answer)
import numpy as np from sklearn.decomposition import PCA # generate gaussian samples (n by 2) rng = np.random.default_rng(4) n = 100 X = rng.standard_normal((n, 2)) @ rng.random((2, 2)) # peel classes apart (+1 and -1 vertically) y = 2 * np.arange(X.shape) // n X[:, 1] = np.where(y, X[:, 1] - 1, X[:, 1] + 1) # project X mu = X.mean(axis=0) X -= mu pca = PCA(n_components=2).fit(X) # explained variance ratios: [0.83267962, 0.16732038] X_pca = pca.transform(X) # reduce dimensionality via PC1 X_pc1 = X_pca[:, ] @ pca.components_[, :] + mu # reduce dimensionality via PC2 X_pc2 = X_pca[:, ] @ pca.components_[, :] + mu
No. PCA targets different target than Linear discriminant analysis (LDA). Both of them actually provide linear transformation, but (rough an quick explanation follows):
- PCA will set the direction of the first axis of the output space in the direction of the maximum variance of the input data (it does not know the concept of classes per se, all samples are treated equally)
- LDA will set the direction of the first axis of the output space in the direction which has (roughly) the best ratio of inter-class variance vs intra-class variance. For that, it needs a concept of classes.