0
$\begingroup$

I'm trying to use GridSearchCV to select the optimal C value in this simple SVM problem. The issue I'm having is that when I run the code the optimal C is chosen to be ridiculously small (~e-18) so that all samples lie within the margin. Even when I alter the samples so that they are easily separable, the optimal C is still on the scale of e-18. Does anyone know why this is happening?

import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.datasets.samples_generator import make_blobs
from sklearn.svm import SVC
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report, confusion_matrix
from sklearn.model_selection import GridSearchCV

X, y = make_blobs(n_samples = 500, centers = 2, random_state = 6,
                  cluster_std = 1.2)

fig = plt.figure()
ax = fig.add_subplot(111)
ax.scatter(X[:,0], X[:,1], c = y, cmap = 'rainbow', s = 30,
           edgecolors = 'white')
ax.set_xlabel(r'$x_1$', fontsize = 20)
ax.set_ylabel(r'$x_2$', fontsize = 20)

svc = SVC(kernel = 'linear')
c_space = np.logspace(-20, 1, 50)
param_grid = {'C': c_space}
svc_cv = GridSearchCV(svc, param_grid, cv = 5)
svc_cv.fit(X, y)
c = svc_cv.best_params_['C']
svc.C = c
svc.fit(X, y)

support_vecs = svc.support_vectors_

x1_min = min(X[:,0])
x1_max = max(X[:,0])
x2_min = min(X[:,1])
x2_max = max(X[:,1])
x1 = np.linspace(x1_min, x1_max, 100)
x2 = np.linspace(x2_min, x2_max, 100)
X1, X2 = np.meshgrid(x1, x2)
points = np.vstack([X1.ravel(), X2.ravel()]).T
boundary = svc.decision_function(points).reshape(X1.shape)
ax.contour(X1, X2, boundary, colors = 'k', levels = [-1, 0, 1],
           linestyles = ['--', '-', '--'])
ax.scatter(support_vecs[:,0], support_vecs[:,1], s = 250, linewidth = 1,
           facecolors = 'none', edgecolors = 'k')

```
$\endgroup$

2 Answers 2

1
$\begingroup$

There are basically two problems:

  1. As Patrick said, your range of C-values doesn't make much sense; but, more importantly
  2. The default metric used by GridSearchCV, the accuracy, is not suitable for what you're trying to do: minimise the number of support vectors while keeping the optimal performance (nor is any other of the metrics provided in sklearn, btw.). The accuracy simply looks whether the classified observations are on the right side of the computed class boundary, and does not care at all whether they are within the margin or not. GridSearchCV simply goes sequentially through your c_space and takes the first value which gives the best accuracy. In your nearly separable case this happens already for a very low C.

So, regarding (1), C scales inversely with the number of points and with the scaling of the data (see this answer for details). This gives you some reasonable range in which you should look for the optimal value. For your data this could be something like:

param_grid = { 'C' : np.logspace(-3, 3, 10) }

Regarding (2), you should define your own scoring metric, e.g. like this:

def margin_accuracy(clf, X_valid, y_true):
  # this is just to ensure that y_true is in {-1, 1}:
  if ~np.any(y_true==-1): y_true = 2*y_true - 1
  N = y_true.shape[0] # the number of samples
  # we compute the false positives and false negatives as the observation
  # on the wrong side of the margin (NOT the class boundary!):
  fp_fn = np.sum(y_true*clf.decision_function(X_valid) < 1)
  return 1 - fp_fn / N # the so modified "accuracy"

and provide this metric as a parameter to GridSearchCV:

svc_cv = GridSearchCV(svc, param_grid, scoring=margin_accuracy, cv=5)

This produces C = 10 and the following classification:

SVC with almost linearly separable classes

Even for a significant overlap in the classes:

X, y = make_blobs(n_samples = 500, centers = 2, random_state = 6,
                  cluster_std = 2.5)

we get a reasonable classifier with C = 46.416:

SVC with a significant class overlap

$\endgroup$
0
$\begingroup$

The question is, what do you expect for a c parameter if its boundaries start at e-20:

c_space = np.logspace(-20, 1, 50)
param_grid = {'C': c_space}

np.logspace

scikit learn already tells you that the c parameter can be definetely higher than 1 or 10 https://scikit-learn.org/stable/auto_examples/svm/plot_rbf_parameters.html

Look at your gamma then you may be able to also get a grasp at why your c is so low (look at the plots of in the link). If gamma and c are e.g. very low, it could be that your data seems linearly seperable.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.