2
$\begingroup$

I am trying to understand which model might work for a given problem before trying the models, I find this case against my knowledge. Please guide what I am missing. I am new to Data Science.

Here is the graph which I got through PCA : enter image description here

Now you can see the boundaries are very much overlapping. The theory for SVM says that this model might work best with overlapping non linear data, which does not seems to be this case.

But still its able to identify all data in test set. So can you provide some clarity on why SVM performing good in this.

So my final results it is below order:

  • Logistic Regression and SVM are same (Accuracy Score : 1.0)
  • Random Forest (Accuracy Score : 0.9680851063829787)
  • KNN (Accuracy Score : 0.925531914893617)

other details :

  • feature set : 40
  • sample data : around 500
$\endgroup$
  • $\begingroup$ when using PCA for data visualization, always report the proportion of variance explained by the two plotted components $\endgroup$ – carlo Jun 3 at 8:09
4
$\begingroup$

When you say 'overlapping' you are focusing on the PCA 2D plot, which is not the original data, but a data projection on reduced feature space.

If logistic regression can achieve 100% accuracy, it means the data is linear separable in original space (with 40 features). It also means the problem is an 'easy' classification problem. This is why most models performs well.

The problem is you are using the wrong way to look at the data.

Because we can only easily see data in 2D, you use PCA and check data in 2D and got the conclusion that data is overlapping. But the truth is data is very separable in original feature space.

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

Its difficult to answer your question as you have not provided enough information on the model parameters, but just by looking at that 2D PCA graph, it will be impossible for any reasonably robust classifier to be able to achieve 100% accuracy without over fitting by only using the first 2 PCs.

I suspect you are using many more features than just 2, your accuracy scores are not holdout scores, and your models are large, hence overfitting.

Depending on the kernel you're using, SVMs can have linear decision boundaries (/w linear kernel) or non-linear decision boundaries (/w non-linear kernel).

A complex enough SVM with enough capacity to over-fit every data point can certainly achieve 100% accuracy.

This is the same with nearly all ML methods.

I recommend you look into cross validation strategies.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.