I am currently using an SVM with a linear kernel to classify my data. There is no error on the training set. I tried several values for the parameter $C$ ($10^{-5}, \dots, 10^2$). This did not change the error on the test set.
Now I
wonder: is this an error caused by the ruby bindings for libsvm I am using
(rb-libsvm) or is this theoretically explainable?
Should the parameter $C$ always change the performance of the classifier?
The C hyperparameter tells the SVM optimization how much you want to avoid misclassifying each training example. For large values of C, the optimization will choose a smaller-margin hyperplane if that hyperplane does a better job of getting all the training points classified correctly. Conversely, a very small value of C will cause the optimizer to look for a larger-margin separating hyperplane, even if that hyperplane misclassifies more points. For very tiny values of C, you should get misclassified examples, often even if your training data is linearly separable.
In a SVM you are searching for two things: a hyperplane with the largest minimum margin, and a hyperplane that correctly separates as many instances as possible. The problem is that you will not always be able to get both things. The c parameter determines how great your desire is for the latter. I have drawn a small example below to illustrate this. To the left you have a low c which gives you a pretty large minimum margin (purple). However, this requires that we neglect the blue circle outlier that we have failed to classify correct. On the right you have a high c. Now you will not neglect the outlier and thus end up with a much smaller margin.
So which of these classifiers are the best? That depends on what the future data you will predict looks like, and most often you don't know that of course. If the future data looks like this:
then the classifier learned using a large c value is best.
On the other hand, if the future data looks like this:
then the classifier learned using a low c value is best.
Depending on your data set, changing c may or may not produce a different hyperplane. If it does produce a different hyperplane, that does not imply that your classifier will output different classes for the particular data you have used it to classify. Weka is a good tool for visualizing data and playing around with different settings for an SVM. It may help you get a better idea of how your data look and why changing the c value does not change the classification error. In general, having few training instances and many attributes make it easier to make a linear separation of the data. Also that fact that you are evaluating on your training data and not new unseen data makes separation easier.
What kind of data are you trying to learn a model from? How much data? Can we see it?
C is essentially a regularisation parameter, which controls the trade-off between achieving a low error on the training data and minimising the norm of the weights. It is analageous to the ridge parameter in ridge regression (in fact in practice there is little difference in performance or theory between linear SVMs and ridge regression, so I generally use the latter - or kernel ridge regression if there are more attributes than observations).
Tuning C correctly is a vital step in best practice in the use of SVMs, as structural risk minimisation (the key principle behind the basic approach) is party implemented via the tuning of C. The parameter C enforces an upper bound on the norm of the weights, which means that there is a nested set of hypothesis classes indexed by C. As we increase C, we increase the complexity of the hypothesis class (if we increase C slightly, we can still form all of the linear models that we could before and also some that we couldn't before we increased the upper bound on the allowable norm of the weights). So as well as implementing SRM via maximum margin classification, it is also implemented by the limiting the complexity of the hypothesis class via controlling C.
Sadly the theory for determining how to set C is not very well developed at the moment, so most people tend to use cross-validation (if they do anything).
The parameter C enforces an upper bound on the norm of the weights. It does not enforce an upper bound. Let's say it does. So, if we choose a large C, we will prefer a larger number of misclassified examples.
$\endgroup$
C is a regularization parameter that controls the trade off between the achieving a low training error and a low testing error that is the ability to generalize your classifier to unseen data.
Consider the objective function of a linear SVM : min |w|^2+C∑ξ. If your C is too large the optimization algorithm will try to reduce |w| as much as possible leading to a hyperplane which tries to classify each training example correctly. Doing this will lead to loss in generalization properties of the classifier. On the other hand if your C is too small then you give your objective function a certain freedom to increase |w| a lot, which will lead to large training error.
The pictures below might help you visualize this.
Most of the answers above are quite good, but let me clarify something for someone like me who had to spent 3 days on understanding the role Parameter C in SVM because of diffrent sources.
In book ISLR(http://faculty.marshall.usc.edu/gareth-james/ISL/) larger C means larger misclassification are allowed which makes margin wider and smaller C means less misclassification is allowed which leads to small margin. Whereas every other resource i have read and in python documentation it is just the opposite.
Actually is ISLR C is defined as the upper bound of the sum of all slack variables.
But in python and other source(https://shuzhanfan.github.io/2018/05/understanding-mathematics-behind-support-vector-machines/#:~:text=In%20terms%20of%20the%20SVM,%2Bb)%E2%88%921%5D.) C is constraints on slack variables.If we set C to positive infinite, we will get the same result as the Hard Margin SVM. On the contrary, if we set C to 0, there will be no constraint anymore, and we will end up with a hyperplane not classifying anything. The rules of thumb are: small values of C will result in a wider margin, at the cost of some misclassifications; large values of C will give you the Hard Margin classifier and tolerates zero constraint violation
scikit-learn.
$\endgroup$
The answers above are excellent. After carefully reading your questions, I found there are 2 important facts we might overlooked.
Given the 2 facts, if C values changes within a reasonable range, the optimal hyperplane will just randomly shifting by a small amount within the margin(the gap formed by the support vectors).
Intuitively, suppose the margin on training data is small, and/or there is no test data points within the margin too, the shifting of the optimal hyperplane within the margin will not affect classification error of the test set.
Nonetheless, if you set C=0, then SVM will ignore the errors, and just try to minimise the sum of squares of the weights(w), perhaps you may get different results on the test set.
C Parameter is used for controlling the outliers — low C implies we are allowing more outliers, high C implies we are allowing fewer outliers.
High C (cost) means the cost of misclassification is increased. This means a flexible kernel will become more squiggly to avoid misclassifying observations in the training set.
If the kernel is to squiggly the model won't generalize well when predicting on new data.
If the kernel is to straight the model won't generalize well when predicting on new data.
