After using PCA to reduce dimensionality, does PCA preserve linear separability for any linearly separable set? Will the data still be linearly separable after the transformation? I am thinking that it does preserve linear separability because the PCA just reduces the dimension and not the relationship between the points in terms of separability. Am I on the right line of thinking?
2
-
3$\begingroup$ The question is ambiguous. It could be "Is there any linearly separable set for which PCA preserves linear separability?" (i.e. is there some such set?) or "Is it the case that for any linearly separable set, no matter which one, PCA preserves linear separability?" (i.e. is this true of every such set). I suspect the latter was intended, in which case just changing "any" to "every" would make it unambiguous. $\endgroup$– Michael HardyMar 6, 2022 at 17:59
-
$\begingroup$ @MichaelHardy, I will fix that accordingly. Thank you for catching the ambiguity $\endgroup$– LVSTMar 8, 2022 at 2:35
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
1
No, it may be that the discriminative information is in the direction of a principal component that explains a relatively small amount of the total variance, and hence gets discarded.
Consider a two-dimensional dataset where the two classes lie in long parallel cigar-shaped elongated Gaussian clusters with a small gap between them. Most of the variance lies along the long axis of the clusters, so the first PC will be in that direction and the second will be orthogonal to it. The data are indistinguishable from the first component, so if you discard the second component, the data will no longer be separable.
answered Mar 4, 2022 at 22:50
-
5$\begingroup$ +1. The simplest example I can find uses the line $y=\epsilon/2$ to separate the points $(\pm1,0)$ from the point $(0,\epsilon)$ for tiny positive $\epsilon.$ Clearly the first principal component must be nearly parallel to the x-axis, thereby projecting the first two points approximately to $\pm 1$ and the second point to $0,$ lying between them on the line: obviously no longer linearly separated. $\endgroup$– whuber ♦Mar 4, 2022 at 23:07
$\begingroup$
$\endgroup$
No, reducing dimensionality with PCA will only maximize variance, which may or may not translate to linear separability.
Here are two visualizations of variance and separability in opposition. In both cases, the discriminative information lies primarily along the low-variance axis, which would get discarded by rote dimensionality reduction.
Minimal example of three data points (adapted from whuber's comment)
Example of parallel, elongated Gaussian clusters (adapted from Dikran's answer)
import numpy as np
from sklearn.decomposition import PCA
# generate gaussian samples (n by 2)
rng = np.random.default_rng(4)
n = 100
X = rng.standard_normal((n, 2)) @ rng.random((2, 2))
# peel classes apart (+1 and -1 vertically)
y = 2 * np.arange(X.shape[0]) // n
X[:, 1] = np.where(y, X[:, 1] - 1, X[:, 1] + 1)
# project X
mu = X.mean(axis=0)
X -= mu
pca = PCA(n_components=2).fit(X) # explained variance ratios: [0.83267962, 0.16732038]
X_pca = pca.transform(X)
# reduce dimensionality via PC1
X_pc1 = X_pca[:, [0]] @ pca.components_[[0], :] + mu
# reduce dimensionality via PC2
X_pc2 = X_pca[:, [1]] @ pca.components_[[1], :] + mu
$\begingroup$
$\endgroup$
No. PCA targets different target than Linear discriminant analysis (LDA). Both of them actually provide linear transformation, but (rough an quick explanation follows):
- PCA will set the direction of the first axis of the output space in the direction of the maximum variance of the input data (it does not know the concept of classes per se, all samples are treated equally)
- LDA will set the direction of the first axis of the output space in the direction which has (roughly) the best ratio of inter-class variance vs intra-class variance. For that, it needs a concept of classes.
