PCA shows overlapping boundaries, then why SVM performs best

Question

I am trying to understand which model might work for a given problem before trying the models, I find this case against my knowledge. Please guide what I am missing. I am new to Data Science.

Here is the graph which I got through PCA :

Now you can see the boundaries are very much overlapping. The theory for SVM says that this model might work best with overlapping non linear data, which does not seems to be this case.

But still its able to identify all data in test set. So can you provide some clarity on why SVM performing good in this.

So my final results it is below order:

• Logistic Regression and SVM are same (Accuracy Score : 1.0)
• Random Forest (Accuracy Score : 0.9680851063829787)
• KNN (Accuracy Score : 0.925531914893617)

other details :

• feature set : 40
• sample data : around 500

Answer 1

When you say 'overlapping' you are focusing on the PCA 2D plot, which is not the original data, but a data projection on reduced feature space.

If logistic regression can achieve 100% accuracy, it means the data is linear separable in original space (with 40 features). It also means the problem is an 'easy' classification problem. This is why most models performs well.

The problem is you are using the wrong way to look at the data.

Because we can only easily see data in 2D, you use PCA and check data in 2D and got the conclusion that data is overlapping. But the truth is data is very separable in original feature space.

Answer 2

Its difficult to answer your question as you have not provided enough information on the model parameters, but just by looking at that 2D PCA graph, it will be impossible for any reasonably robust classifier to be able to achieve 100% accuracy without over fitting by only using the first 2 PCs.

I suspect you are using many more features than just 2, your accuracy scores are not holdout scores, and your models are large, hence overfitting.

Depending on the kernel you're using, SVMs can have linear decision boundaries (/w linear kernel) or non-linear decision boundaries (/w non-linear kernel).

A complex enough SVM with enough capacity to over-fit every data point can certainly achieve 100% accuracy.

This is the same with nearly all ML methods.

I recommend you look into cross validation strategies.