1
$\begingroup$

I have a multi-class (10 classes) classification problem. I am using one-vs-rest SVM modeling with sklearn.svm.SVC.

I want to know whether my model is over-fitting.

For train set accuracy is 100% and for test set its 97%+.

To analyze the model, I did the following:

  1. I plotted probability (using predict_proba(), that works as expalined here).

In all the following plots, colors and legends represent the 10 classes, with labels ranging in 0-9.

The figure below shows the probabilities of true class for the train set instances: Probabilities for the train set instances. y-axis: probability, x-axis: indexes of the samples

The following figure shows the probabilities of true class for the test set instances: Probabilities for the test set instances. y-axis: probability

Observation: The train set contains very few instances with probability < 0.95. Whereas test set has many instances with low probabilities for true class. Should this be treated as over-fitting of the model?

  1. Similarly, I plotted distance (using decision_function()) from separating hyperplane.

Train set distance of true class from separating hyperplane: Train set distance from separating hyperplane. y-axis: distance

Test set distance of true class from separating hyperplane: Test set distance from separating hyperplane

  1. Then I compared the probabilities of true class vs best of the rest class (which I call as best alternate class). X-label represents true class probability. And y-label represents the probability of its corresponding best alternate class.

Probability comparison on train set: Probability comparison on train set

Probability comparison on test set: Probability comparison on test set

  1. Similarly, I compared the distances from the separating hyperplane. X-label and y-label conventions are similar to the ones above for comparing probabilities.

Distance comparison on train set: Distance comparison on train set

Distance comparison on test set: Distance comparison on test set

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.