2
$\begingroup$

In soft margin SVM with Gaussian kernel, How will the decision boundary change, if $ \sigma $ is increased? I means what will be happened in this case. i.e:

$(1)$ Dose the number of misclassified data increase?

$(2)$ How about $C$? If $C$ is increased, Dose the number of misclassified data increase?

In my opinion, When $ \sigma $ is increased, the number of misclassified data will be increased.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Not necessarily. These parameters control how "complex" or "overfit" the model can be. If your model is too simple or too complex, misclassification will increase. There is some sweet spot of parameters for your data and problem, which you should try to find experimentally by measuring misclassification rate on a holdout set (a random sample of your data that is excluded from the SVM).

This article has nice visualizations for an RBF SVM showing what an underfit or overfit model looks like, but the concept is similar for any kernel. (I think sigma works opposite from gamma, where a bigger sigma regularizes, i.e., makes the model less complex).

Improve this answer
$\endgroup$
4
  • $\begingroup$ But when σ is increased, the decision boundary will be smoother. So the number of misclassified will be increased. It's not true all time? $\endgroup$
    – bethany
    Jan 8, 2021 at 16:44
  • $\begingroup$ So to be precise about it, you are correct that the number of in sample misclassified count will always increase. But that doesn't mean the model is better, and in fact a smoother model will often generalize better to a holdout set, meaning that if you were to resample data from the same underlying distribution (e.g., apply your model in the real world to new data), the smoother model will still work well, whereas the model that was overfit to your first sample will suddenly have very high misclassification. $\endgroup$
    – Paul Fornia
    Jan 8, 2021 at 16:51
  • $\begingroup$ The wiki on overfitting has some really nice visuals. In fig 1, the green line is like a small sigma. "Perfect" accuracy, but nonsensical. The black line is smoother, "misses" more points in the training data, but is actually a superior model because it would miss fewer points if applied to another similar dataset. $\endgroup$
    – Paul Fornia
    Jan 8, 2021 at 16:56
  • $\begingroup$ Sorry, typo in my first comment. meant to say "But that doesn't mean the model is worse". $\endgroup$
    – Paul Fornia
    Jan 8, 2021 at 16:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.