The best way to understand the role of the hyperparameter $C$ is to think in term of robustness of your classifier. In short, $C$ is the hyperparameter that you will have to tune to find the optimal trade off between two competing objectives, which are 1) to make sure that your classifier performs well on your training data and 2) that it will also generalize well on unseen data.
Let's assume that you begin by training a "classical" SVM (i.e. without the regularization terms $C$ and $\xi_i$). If your training data set is well behaved (i.e. without outliers and linearly separable), then end result should look like this:
with the solid line being the classifier. However, if you train the same kind of SVM on a data set containing outliers, the result will be the following:
Here, the red point is an outlier, and using the unregularized objective function will lead to the classifier drawn as a solid line (instead of the expected classifier represented by a dashed line that we would have obtained, had we not considered the outlier). The classifier performs well on your training set (since all data points are correctly classified), but will poorly generalize on new data points (any new point falling into the blue area will be missclassified). This classifier is said to overfit or to lack robustness.
One way to avid this pitfall is to modify your objective function so as to allow some of your training points to be missclassified (i.e. to have a margin of $1-\xi_i$ instead of $1$). But we don't want too many of these points to be missclassified (otherwise the whole point of training a SVM to perform classification disappears). We thus have two competing objectives:
- On the one hand, like in the classical case, we want the $\xi_i$ not to be too high, to make sure the boundary is violated as little as possible. This is represented by adding the $\sum_i \xi_i$ term in the objective function;
- On the other hand, we also want to ensure the robustness of the classifier. The constraints are thus relaxed and become $y^{(i)}(w^T x^{(i)} + b) \geq 1-\xi_i$ instead of $y^{(i)}(w^T x^{(i)} + b) \geq 1$.
The hyperparameter $C$ determines how much you care about the respective objectives. If you set $C$ to be very high, then the only way to minimize your objective function will be to set all the $\xi_i$ to $0$. You are thus back to the original case in which the constraints are not relaxed. In contrary, if you care about the robustness of your classifier, you should set $C$ to a small value (to allow the $\xi_i$ to take whichever values they want).
Finding the optimal value for $C$ can be done by performing cross validation. If you obtain a very low training error, but a high test error (which is a telling sign of overfitting), you should consider increasing the value of $C$ and redo the whole tuning.
Even if you know a priori that your data set is linearly separable, adding the regularization terms $\xi_i$ is still a good idea, as it will allow your classifier to generalize well in case of outliers.
