I have a multi-class (10 classes) classification problem. I am using one-vs-rest SVM modeling with sklearn.svm.SVC.
I want to know whether my model is over-fitting.
For train set accuracy is 100% and for test set its 97%+.
To analyze the model, I did the following:
In all the following plots, colors and legends represent the 10 classes, with labels ranging in 0-9.
Observation: The train set contains very few instances with probability < 0.95. Whereas test set has many instances with low probabilities for true class. Should this be treated as over-fitting of the model?
- Similarly, I plotted distance (using decision_function()) from separating hyperplane.
- Then I compared the probabilities of true class vs best of the rest class (which I call as best alternate class). X-label represents true class probability. And y-label represents the probability of its corresponding best alternate class.
- Similarly, I compared the distances from the separating hyperplane. X-label and y-label conventions are similar to the ones above for comparing probabilities.