# What is the influence of C in SVMs with linear kernel?

I am currently using an SVM with a linear kernel to classify my data. There is no error on the training set. I tried several values for the parameter $C$ ($10^{-5}, \dots, 10^2$). This did not change the error on the test set.

Now I wonder: is this an error caused by the ruby bindings for libsvm I am using (rb-libsvm) or is this theoretically explainable?

Should the parameter $C$ always change the performance of the classifier?

• Just a comment, not an answer: Any program that minimizes a sum of two terms, such as $|w|^2 + C \sum{ \xi_i },$ should (imho) tell you what the two terms are at the end, so that you can see how they balance. (For help on computing the two SVM terms yourself, try asking a separate question. Have you looked at a few of the worst-classified points ? Could you post a problem similar to yours ?) Jul 13, 2012 at 20:02

The C hyperparameter tells the SVM optimization how much you want to avoid misclassifying each training example. For large values of C, the optimization will choose a smaller-margin hyperplane if that hyperplane does a better job of getting all the training points classified correctly. Conversely, a very small value of C will cause the optimizer to look for a larger-margin separating hyperplane, even if that hyperplane misclassifies more points. For very tiny values of C, you should get misclassified examples, often even if your training data is linearly separable.

• OK, I understand that C determines the influence of the misclassification on the objective function. The objective function is the sum of a regularization term and the misclassification rate (see en.wikipedia.org/wiki/Support_vector_machine#Soft_margin). When I change C, this does not have any effect on the minimum of my objective function. Could that mean that the regularization term is always very small?
– alfa
Jun 24, 2012 at 12:31
• I would suggest trying a wider range of C values, maybe 10^[-5,...,5], or more if the optimization is fast on your dataset, to see if you get something that looks more reasonable. Both the training error and the value of the minimum cost should change as C is varied. Also, is the scale of your data extreme? In general, an optimal C parameter should be larger when you scale down your data, and vice versa, so if you have very small values for features, make sure to include very large values for the possible C values. If none of the above helps, I'd guess the problem is in the ruby bindings Jun 24, 2012 at 19:59
• changing the balanced accuracy from 0.5 (just guessing) to 0.86 doesn't sound like a marginal influence to me. It would be a good idea to investigate a finer grid of values for C as Marc suggests, but the results you gave given seem to be fairly normal behaviour. One might expect the error to go back up again as C tends to infinity due to over-fitting, but that doesn't seem to much of a problem in this case. Note that if you are really interested in balanced error and your training set doesn't have a 50:50 split, then you may be able to get better results... Jun 25, 2012 at 12:58
• ... by using different values of C for patterns belonging to the positive and negative classes (which is asymptotically equivalent to resampling the data to change the proportion of patterns belonging to each class). Jun 25, 2012 at 12:59
• I think it is possible that once you get to C=10^0 the SVM is already classifying all of the training data correctly, and none of the support vectors are bound (the alpha is equal to C) in that case making C bigger has no effect on the solution. Jun 26, 2012 at 12:26

In a SVM you are searching for two things: a hyperplane with the largest minimum margin, and a hyperplane that correctly separates as many instances as possible. The problem is that you will not always be able to get both things. The c parameter determines how great your desire is for the latter. I have drawn a small example below to illustrate this. To the left you have a low c which gives you a pretty large minimum margin (purple). However, this requires that we neglect the blue circle outlier that we have failed to classify correct. On the right you have a high c. Now you will not neglect the outlier and thus end up with a much smaller margin.

So which of these classifiers are the best? That depends on what the future data you will predict looks like, and most often you don't know that of course. If the future data looks like this:

then the classifier learned using a large c value is best.

On the other hand, if the future data looks like this:

then the classifier learned using a low c value is best.

Depending on your data set, changing c may or may not produce a different hyperplane. If it does produce a different hyperplane, that does not imply that your classifier will output different classes for the particular data you have used it to classify. Weka is a good tool for visualizing data and playing around with different settings for an SVM. It may help you get a better idea of how your data look and why changing the c value does not change the classification error. In general, having few training instances and many attributes make it easier to make a linear separation of the data. Also that fact that you are evaluating on your training data and not new unseen data makes separation easier.

What kind of data are you trying to learn a model from? How much data? Can we see it?

• I did not touch the data for more than 3 years now. It is very high-dimensional and noisy and I am not allowed to publish it. The question has been answered already but I think your visualization is very good and intuitive.
– alfa
Sep 2, 2015 at 15:14
• awesome. do you have such explanations for gama value too ? Oct 17, 2015 at 18:33
• The gamma parameter is used for the Gaussian kernel function. The kernel functions can be seen as an efficient way to transform your original features into another space, where a separating hyperplane in the new feature space does not have to be linear in the original feature space. For instance, the two dimensional position of a data point in the original feature space could be used to calculate a new feature representing the distance to some marker on a map. With this new feature, a non-linear classifier (in original space) can be made which decision boundary forms a circle around the marker Oct 23, 2015 at 12:57
• @KentMuntheCaspersen isn't your explanation of C incorrect? It's the opposite of what it says in the book "Introduction to Statistical Learning". Feb 6, 2017 at 17:40
• @diugalde can you quote from the book what exactly differs from my explanation? I always think of c as the cost of misclassification (easy to remember by c in classification). In that way higher c means high cost of misclassification, leading to the algorithm trying to perfectly separate all data points. With outliers this is not always possible or wont always lead to a good general result, which is a good reason for lowering / introducing c. May 17, 2017 at 9:16

C is essentially a regularisation parameter, which controls the trade-off between achieving a low error on the training data and minimising the norm of the weights. It is analageous to the ridge parameter in ridge regression (in fact in practice there is little difference in performance or theory between linear SVMs and ridge regression, so I generally use the latter - or kernel ridge regression if there are more attributes than observations).

Tuning C correctly is a vital step in best practice in the use of SVMs, as structural risk minimisation (the key principle behind the basic approach) is party implemented via the tuning of C. The parameter C enforces an upper bound on the norm of the weights, which means that there is a nested set of hypothesis classes indexed by C. As we increase C, we increase the complexity of the hypothesis class (if we increase C slightly, we can still form all of the linear models that we could before and also some that we couldn't before we increased the upper bound on the allowable norm of the weights). So as well as implementing SRM via maximum margin classification, it is also implemented by the limiting the complexity of the hypothesis class via controlling C.

Sadly the theory for determining how to set C is not very well developed at the moment, so most people tend to use cross-validation (if they do anything).

• OK, I think I understand the meaning of C now. :)
– alfa
Jun 25, 2012 at 19:20
• But if C is a regularization parameter, why does a high C increase overfitting, when generally speaking regularization is done to mitigate overfitting, i.e., by creating a more general model? Feb 26, 2016 at 14:58
• C is a regularisation parameter, but it is essentially attached to the data misfit term (the sum of the slack variables) rather than the regularisation term (the margin bit), so a larger value of C means less regularisation, rather than more. Alternatively you can view the usual representation of the rgularisation parameter as 1/C. Feb 26, 2016 at 15:36
• This expression is wrong: The parameter C enforces an upper bound on the norm of the weights. It does not enforce an upper bound. Let's say it does. So, if we choose a large C, we will prefer a larger number of misclassified examples.
– ARAT
May 21, 2020 at 14:55
• @ARAT if you look at the dual formulation for the optimisation problem for the SVM, there is a box constraint that prevents the dual parameters from exceeding C. Putting an upper bound on the dual parameters will put an upper bound on the norm of the primal weights as well. This imposes a heirarchical set of hypotheses classes, which is essentially how SRM works. May 22, 2020 at 7:22

C is a regularization parameter that controls the trade off between the achieving a low training error and a low testing error that is the ability to generalize your classifier to unseen data.

Consider the objective function of a linear SVM : min |w|^2+C∑ξ. If your C is too large the optimization algorithm will try to reduce |w| as much as possible leading to a hyperplane which tries to classify each training example correctly. Doing this will lead to loss in generalization properties of the classifier. On the other hand if your C is too small then you give your objective function a certain freedom to increase |w| a lot, which will lead to large training error.

• I don't really understand your plots. Can you explain it?
– alfa
Mar 20, 2015 at 10:14
• @alfa : My intent for showing the plots was : 1) If C is too large (plot 1) , then your classifier will over fit ,i.e. it'll try to classify each training data point accurately. Plot 1 shows almost all training points being classified correctly. 2)On the other hand if C is too less (plot 2) , then your classifier will under fit. Plot 2 shows the under fit classifier. It does not segregate the points into their respective classes. Hope this helps. Apr 10, 2015 at 11:09
• That means that your x- and y-axes show two different features. The labels "length of dataset" and "Sum of means" are a little bit confusing?
– alfa
Apr 10, 2015 at 13:46
• It would be interesting to see how the right choice for C helps in both cases.
– alfa
Apr 10, 2015 at 13:49
• I think it is not obvious to see that C=10000000 is a bad choice and I think the dataset is not the right one to demonstrate that. Maybe a dataset with only a few outliers on the wrong side of the separating hyperplane would be better?
– alfa
Apr 12, 2015 at 12:28

Most of the answers above are quite good, but let me clarify something for someone like me who had to spent 3 days on understanding the role Parameter C in SVM because of diffrent sources.

In book ISLR(http://faculty.marshall.usc.edu/gareth-james/ISL/) larger C means larger misclassification are allowed which makes margin wider and smaller C means less misclassification is allowed which leads to small margin. Whereas every other resource i have read and in python documentation it is just the opposite.

Actually is ISLR C is defined as the upper bound of the sum of all slack variables.

But in python and other source(https://shuzhanfan.github.io/2018/05/understanding-mathematics-behind-support-vector-machines/#:~:text=In%20terms%20of%20the%20SVM,%2Bb)%E2%88%921%5D.) C is constraints on slack variables.If we set C to positive infinite, we will get the same result as the Hard Margin SVM. On the contrary, if we set C to 0, there will be no constraint anymore, and we will end up with a hyperplane not classifying anything. The rules of thumb are: small values of C will result in a wider margin, at the cost of some misclassifications; large values of C will give you the Hard Margin classifier and tolerates zero constraint violation

– Sycorax
Sep 18, 2020 at 15:26
• Appreciate someone adding this clarification. I ran into the same issue in the difference between ISLR and other sources, e.g. scikit-learn. Mar 20, 2021 at 23:29

The answers above are excellent. After carefully reading your questions, I found there are 2 important facts we might overlooked.

1. You are using linear kernel
2. Your training data is linearly separable, since "There is no error on the training set".

Given the 2 facts, if C values changes within a reasonable range, the optimal hyperplane will just randomly shifting by a small amount within the margin(the gap formed by the support vectors).

Intuitively, suppose the margin on training data is small, and/or there is no test data points within the margin too, the shifting of the optimal hyperplane within the margin will not affect classification error of the test set.

Nonetheless, if you set C=0, then SVM will ignore the errors, and just try to minimise the sum of squares of the weights(w), perhaps you may get different results on the test set.

C Parameter is used for controlling the outliers — low C implies we are allowing more outliers, high C implies we are allowing fewer outliers.

High C (cost) means the cost of misclassification is increased. This means a flexible kernel will become more squiggly to avoid misclassifying observations in the training set.

If the kernel is to squiggly the model won't generalize well when predicting on new data.

If the kernel is to straight the model won't generalize well when predicting on new data.