I was reading the Hands on ML book and I'm on the SVM and Logistic Regression chapters. I started looking up more on these algorithms and apparently they are "linear" classifiers i.e the decision boundary is linear (The classifier needs the inputs to be linearly separable.)

Now in the book it is mentioned that since in most of the cases data is not linearly separable, we have to increase the dimensions of the features to make it linearly separable.

But is it always true that there is some transformation to convert every non-linearly separable data set into a linearly separable one? If not, what would be an example of such a data set where this is impossible?

  • 2
    $\begingroup$ You can always make a dataset linearly separable in a trivial (and unhelpful) way: associate a unique real number to each cluster in the dataset and adjoin that number as a new coordinate to all observations. There are other methods that don't require you to know the classifications at the outset, but they typically require many more dimensions. $\endgroup$ – whuber Jul 30 at 15:26
  • 1
    $\begingroup$ Some useful discussion in the context of SVMs: stats.stackexchange.com/questions/94003/… $\endgroup$ – Sycorax Jul 31 at 17:32

In theory, it is always possible to make any arbitrary dataset linearly separable in higher dimensions. In fact, you ideally only need to add one additional dimension to do so, which is a dimension that represents your true class labels. No matter what the data looks like in the other dimensions, if you have a way to add a dimension that represents the true class values, you can linearly separate on that dimension and perfectly recover the true classes. The only time it's impossible to add a dimension like this is if you have two identical samples with different classes, since there will be no deterministic way to map them to different classes given only the feature data.

Otherwise, this mapping is always possible in theory, but in practice, it's usually difficult to come up with a way to generate that extra dimension of class labels which is generalizable and not overfit. A simple transformation is to look at all your datapoints, and just assign the true class as the value on your new dimension, but this method completely fails to generalize to points not in the original data. It's trivial to overfit the mapping to linearly separate the training data, but it's much more difficult to find a mapping that will accurately separate data you didn't train on.

| cite | improve this answer | |
  • 1
    $\begingroup$ Two dimensions is usually enough? I can't recommend taking this answer seriously. Do some basic reading on machine learning. The part about about generalization and overfitting is correct. $\endgroup$ – user255758 Aug 1 at 1:00
  • $\begingroup$ @Jason_93 What I've described is basically the whole point of supervised machine learning - to find a mapping from a set of input features to a target feature. If you can find a prediction output that has an AUC of 1 (it perfectly predicts the target), you have effectively discovered an additional dimension which makes the dataset linearly separable. $\endgroup$ – Nuclear Hoagie Aug 1 at 3:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.