Error distribution of support vector machine

Question

I am recently learning to use support vector machine as classification. I have some question about it and hope that it is not a dumb question.

As far as I know, For multinomial logistic model for classification, the error terms is assumed to be i.i.d double-exponential distribution. For probit model, it is assumed to be normal distribution.

How about support vector machine? Since we assume the data is linearly separable, does the error distribution related to the distance between the hyperplane and the support vector?

The margin $M = \frac{1}{||\beta||}$

where the hyperplane is $\{x: f(x) = x^T \beta + \beta_0 = 0\}$ and the decision rule for class $\{-1,1\}$ is simply $sign(x^T \beta + \beta_0)$.

and I think somehow the "variance" of the error is controlled by the cost and I can use it to control the bias-variance tradeoff?

Answer 1

For the hard-margin support vector machine, there is no real means of controlling the bias-variance trade-off. For that you need to use the soft-margin formulation, where the bias-variance trade-off is controlled by the regularisation parameter, $C$.

The leave-one-out error is bounded by a quantity that depends on the margin (see the radius-margin bound and the span bound). The SVM is essentially an approximate implementation of an upper bound on the generalisation error, which is defined in terms of the margin.

However, I suspect that the reason the SVM works well in practice on most occasions is because it encourages the user to at least think about avoiding over-fitting by tuning the regularisation parameters. The same good performance is often just as easily obtained using [kernel] ridge regression, which suggests the hinge loss doesn't make a great deal of difference (average case).