# Will non-linear data always become linear in high dimension?

I was reading the Hands on ML book and I'm on the SVM and Logistic Regression chapters. I started looking up more on these algorithms and apparently they are "linear" classifiers i.e the decision boundary is linear (The classifier needs the inputs to be linearly separable.)

Now in the book it is mentioned that since in most of the cases data is not linearly separable, we have to increase the dimensions of the features to make it linearly separable.

But is it always true that there is some transformation to convert every non-linearly separable data set into a linearly separable one? If not, what would be an example of such a data set where this is impossible?

• You can always make a dataset linearly separable in a trivial (and unhelpful) way: associate a unique real number to each cluster in the dataset and adjoin that number as a new coordinate to all observations. There are other methods that don't require you to know the classifications at the outset, but they typically require many more dimensions.
– whuber
Commented Jul 30, 2020 at 15:26
• Some useful discussion in the context of SVMs: stats.stackexchange.com/questions/94003/…
– Sycorax
Commented Jul 31, 2020 at 17:32

In theory, it is always possible to make any arbitrary dataset linearly separable in higher dimensions. In fact, you ideally only need to add one additional dimension to do so, which is a dimension that represents your true class labels. No matter what the data looks like in the other dimensions, if you have a way to add a dimension that represents the true class values, you can linearly separate on that dimension and perfectly recover the true classes. The only time it's impossible to add a dimension like this is if you have two identical samples with different classes, since there will be no deterministic way to map them to different classes given only the feature data.

Otherwise, this mapping is always possible in theory, but in practice, it's usually difficult to come up with a way to generate that extra dimension of class labels which is generalizable and not overfit. A simple transformation is to look at all your datapoints, and just assign the true class as the value on your new dimension, but this method completely fails to generalize to points not in the original data. It's trivial to overfit the mapping to linearly separate the training data, but it's much more difficult to find a mapping that will accurately separate data you didn't train on.

• @Jason_93 What I've described is basically the whole point of supervised machine learning - to find a mapping from a set of input features to a target feature. If you can find a prediction output that has an AUC of 1 (it perfectly predicts the target), you have effectively discovered an additional dimension which makes the dataset linearly separable. Commented Aug 1, 2020 at 3:57
• Hello, I didn't understand what you meant by " make any arbitrary dataset linearly separable ...". From what I've seen before, we talk about linear separability of two sets of data lying in a certain dimensions. But here you're just talking about one dataset, so I asked. Commented Nov 18, 2020 at 13:44
• @Mathmath I'm talking about any possible (arbitrary) dataset that contains 2 or more classes - the number or type of features doesn't matter. In this original dimensional space, you may or may not be able to linearly separate the classes. But there will always exist a mapping where you can add just one more dimension which will make the classes linearly separable. This is basically the goal of many supervised machine learning methods - take a set of features and produce an output (i.e. a new feature) that discriminates between the classes. Commented Nov 18, 2020 at 14:38
• Thanks! Can you be a bit more mathematically specific on "...there will always exist a mapping where you can add just one more dimension...(which will make the classes linearly separable)"? Say we've a data set in $R^d$ consisting of two classes $C_1, C_2 \subset R^d$. Assume that $C_1, C_2$ aren't linearly separable in $R^d.$. Are you saying that: then there exists a map $F: R^d \to R^{d+1}$ so that even if two classes weren't linearly separable in $R^d,$ after taking $F$-images, the new $(d+1)$-dimensional sets $F(C_1), F(C_2) \subset R^{d+1}$ will be linearly separable? Cite the theorem? Commented Nov 18, 2020 at 14:45
• @Mathmath Cover's Theorem deals with the notion of separability in high-dimensional space, this is also related to non-linear kernel functions which can be used to map non-linearly separable data into a higher dimensional space where it is linearly separable. Commented Nov 18, 2020 at 15:33

Yes, you can always linearly separate finite dimensional subsets by adding a dimension.

Proposition: If $$X_0$$ and $$X_1$$ are disjoint subsets of $$\mathbb{R}^n$$, then there exists function $$f:\mathbb{R^n}\rightarrow\mathbb{R}^{n+1}$$ such that $$f(X_0)$$ and $$f(X_1)$$ are linearly separable.

Proof: Define $$f$$ as follows:

$$f(x) = (x, 0),$$ for $$x\in X_0$$,

$$f(x) = (x, 1),$$ for $$x\in X_1$$,

and $$f(x) = (x, y)$$ where $$y$$ is arbitrary for $$x$$ elsewhere, e.g., $$y$$ is a realization of a Bernoilli random variable (coin toss) taking values of $$0$$ or $$1$$.

Recall the definition of linear separability of disjoint subsets $$A$$ and $$B$$ in vector space $$U$$ with inner product $$\langle \cdot,\cdot\rangle$$: there exists vector $$v\in U$$ and real $$t$$ such that $$\langle a,v\rangle < t$$ and $$\langle b,v\rangle > t$$ for all $$a \in A$$ and $$b \in B$$.

Let $$A=f(X_0)$$, $$B=f(X_1)$$, $$v=(\vec{0},1)$$ and $$t=1/2$$ in the above definition:

$$\langle (x,0), (\vec{0},1) \rangle = 0 < 1/2 \qquad \forall (x,0) \in f(X_0)$$

and

$$\langle (x,1), (\vec{0},1) \rangle = 1 > 1/2 \qquad \forall (x,1) \in f(X_1).$$

Therefore, hyperplane $$\{ (x, 1/2) : x \in \mathbb{R}^n\} \subset \mathbb{R}^{n+1}$$ linearly separates $$f(X_0)$$ and $$f(X_1)$$. QED

Unfortunately, this specific construction will not generalize well, as @Nuclear Hoagie pointed out.