Will non-linear data always become linear in high dimension?

Question

I was reading the Hands on ML book and I'm on the SVM and Logistic Regression chapters. I started looking up more on these algorithms and apparently they are "linear" classifiers i.e the decision boundary is linear (The classifier needs the inputs to be linearly separable.)

Now in the book it is mentioned that since in most of the cases data is not linearly separable, we have to increase the dimensions of the features to make it linearly separable.

But is it always true that there is some transformation to convert every non-linearly separable data set into a linearly separable one? If not, what would be an example of such a data set where this is impossible?

Answer 1

In theory, it is always possible to make any arbitrary dataset linearly separable in higher dimensions. In fact, you ideally only need to add one additional dimension to do so, which is a dimension that represents your true class labels. No matter what the data looks like in the other dimensions, if you have a way to add a dimension that represents the true class values, you can linearly separate on that dimension and perfectly recover the true classes. The only time it's impossible to add a dimension like this is if you have two identical samples with different classes, since there will be no deterministic way to map them to different classes given only the feature data.

Otherwise, this mapping is always possible in theory, but in practice, it's usually difficult to come up with a way to generate that extra dimension of class labels which is generalizable and not overfit. A simple transformation is to look at all your datapoints, and just assign the true class as the value on your new dimension, but this method completely fails to generalize to points not in the original data. It's trivial to overfit the mapping to linearly separate the training data, but it's much more difficult to find a mapping that will accurately separate data you didn't train on.

Answer 2

Yes, you can always linearly separate finite dimensional subsets by adding a dimension.

Proposition: If $X_0$ and $X_1$ are disjoint subsets of $\mathbb{R}^n$, then there exists function $f:\mathbb{R^n}\rightarrow\mathbb{R}^{n+1}$ such that $f(X_0)$ and $f(X_1)$ are linearly separable.

Proof: Define $f$ as follows:

$f(x) = (x, 0), $ for $x\in X_0$,

$f(x) = (x, 1), $ for $x\in X_1$,

and $f(x) = (x, y)$ where $y$ is arbitrary for $x$ elsewhere, e.g., $y$ is a realization of a Bernoilli random variable (coin toss) taking values of $0$ or $1$.

Recall the definition of linear separability of disjoint subsets $A$ and $B$ in vector space $U$ with inner product $\langle \cdot,\cdot\rangle$: there exists vector $v\in U$ and real $t$ such that $\langle a,v\rangle < t$ and $\langle b,v\rangle > t$ for all $a \in A$ and $b \in B$.

Let $A=f(X_0)$, $B=f(X_1)$, $v=(\vec{0},1)$ and $t=1/2$ in the above definition:

$$ \langle (x,0), (\vec{0},1) \rangle = 0 < 1/2 \qquad \forall (x,0) \in f(X_0)$$

and

$$ \langle (x,1), (\vec{0},1) \rangle = 1 > 1/2 \qquad \forall (x,1) \in f(X_1).$$

Therefore, hyperplane $\{ (x, 1/2) : x \in \mathbb{R}^n\} \subset \mathbb{R}^{n+1}$ linearly separates $f(X_0)$ and $f(X_1)$. QED

Unfortunately, this specific construction will not generalize well, as @Nuclear Hoagie pointed out.