I have a multi-class (10 classes) classification problem. I am using one-vs-rest SVM modeling with sklearn.svm.SVC.

I want to know whether my model is over-fitting.

For train set accuracy is 100% and for test set its 97%+.

To analyze the model, I did the following:

  1. I plotted probability (using predict_proba(), that works as expalined here).

In all the following plots, colors and legends represent the 10 classes, with labels ranging in 0-9.

The figure below shows the probabilities of true class for the train set instances: Probabilities for the train set instances. y-axis: probability, x-axis: indexes of the samples

The following figure shows the probabilities of true class for the test set instances: Probabilities for the test set instances. y-axis: probability

Observation: The train set contains very few instances with probability < 0.95. Whereas test set has many instances with low probabilities for true class. Should this be treated as over-fitting of the model?

  1. Similarly, I plotted distance (using decision_function()) from separating hyperplane.

Train set distance of true class from separating hyperplane: Train set distance from separating hyperplane. y-axis: distance

Test set distance of true class from separating hyperplane: Test set distance from separating hyperplane

  1. Then I compared the probabilities of true class vs best of the rest class (which I call as best alternate class). X-label represents true class probability. And y-label represents the probability of its corresponding best alternate class.

Probability comparison on train set: Probability comparison on train set

Probability comparison on test set: Probability comparison on test set

  1. Similarly, I compared the distances from the separating hyperplane. X-label and y-label conventions are similar to the ones above for comparing probabilities.

Distance comparison on train set: Distance comparison on train set

Distance comparison on test set: Distance comparison on test set


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.