# Why can't scikit-learn SVM solve two concentric circles?

Consider the following dataset (code for generating it is at the bottom of the post):

Running the following code:

from sklearn.svm import SVC
model_2 = SVC(kernel='rbf', degree=2, gamma='auto', C=100)
model_2.fit(X_train, y_train)
print('accuracy (train): %5.2f'%(metric(y_train, model_2.predict(X_train))))
print('accuracy (test): %5.2f'%(metric(y_test, model_2.predict(X_test))))
print('Number of support vectors:', sum(model_2.n_support_))


I get the following output:

accuracy (train):  0.64
accuracy (test):  0.26
Number of support vectors: 55


I also tried with varying degrees of polynomial kernel and got more or less the same results.

So why does it do such a poor job. I've just learned about SVM and I would have thought that a polynomial kernel of 2nd degree could just project these points onto a paraboloid and the result would be linearly separable. Where am I going wrong here?

Reference: The starter code for the snippets in this post comes from this course

Code for generating data:

np.random.seed(0)
data, labels = sklearn.datasets.make_circles()
idx = np.arange(len(labels))
np.random.shuffle(idx)
# train on a random 2/3 and test on the remaining 1/3
idx_train = idx[:2*len(idx)//3]
idx_test = idx[2*len(idx)//3:]
X_train = data[idx_train]
X_test = data[idx_test]

y_train = 2 * labels[idx_train] - 1  # binary -> spin
y_test = 2 * labels[idx_test] - 1

scaler = sklearn.preprocessing.StandardScaler()
normalizer = sklearn.preprocessing.Normalizer()

X_train = scaler.fit_transform(X_train)
X_train = normalizer.fit_transform(X_train)

X_test = scaler.fit_transform(X_test)
X_test = normalizer.fit_transform(X_test)
plt.figure(figsize=(6, 6))
plt.subplot(111)
plt.scatter(data[labels == 0, 0], data[labels == 0, 1], color='navy')
plt.scatter(data[labels == 1, 0], data[labels == 1, 1], color='c')
$$$$

• Thanks for looking into it. The original code used the default value of C (didn't specify it). That was me just playing around with it to try and illicit different results. Jun 17 '20 at 12:19

1. All the preprocessing should be done using training set's fitted values:

X_test = scaler.transform(X_test)
X_test = normalizer.transform(X_test)

2. degree is a hyperparameter for polynomial kernel and is ignored if the kernel is not poly:

model_2 = SVC(kernel='poly', degree=2, gamma='auto', C=100)


OR

model_2 = SVC(kernel='rbf', gamma='auto', C=100)

3. While debugging, print the final dataset after going through preprocessing to see if you've destroyed the dataset:

Do not blindly implement preprocessing. Remove the normalisation step because it just sabotages the dataset. You'll have 100% accuracy.

• I guess another tip is to not blindly follow online course code even if it seems to come from an "official" source... Thanks! Jun 17 '20 at 12:23
• +1 for detailed and fast answer! Jun 17 '20 at 12:25
• very interesting to see we cannot blindly apply pre-processing. i think many people things pre-processing and scale data is always better... Jun 17 '20 at 12:33
• yeah, almost nothing is always better. In this specific toy example, it is the scale that differentiates the classes. Jun 17 '20 at 12:35
• (+1): Might help to explain what normalizer.fit_transform does: "Each sample (i.e. each row of the data matrix) with at least one non zero component is rescaled independently of other samples so that its norm (l1, l2 or inf) equals one". The L2-norm is the default, so it precisely places each pair of observations on the unit circle. Jun 17 '20 at 21:19

@gunes has a very good answer: degree is for poly, and rbf is controlled by gamma and C. In general, it is not surprising to see the default parameter does not work well.

If you change your code

model_2 = SVC(kernel='rbf', gamma=1000, C=100)`

You will see 100% on training but 56% on testing.

The reason is As @gunes mentioned the pre-processing changed the data. this also tells us RBF kernel is pretty powerful that can overfit training data pretty well.

• (+1) Yeah, I've never mentioned the hyper-parameter choice part since it was pretty specified in the question. Jun 17 '20 at 12:30

The answer is very simple and very short. Because you attempt to make a support vector machine create something that is impossible, there is no support vectors that will constrain to only those two circles.

• That's true for a linear SVM, but this should be a cakewalk when using a radial (i.e., circular) basis function--there's very clearly a circle that separates the two classes! Jun 18 '20 at 0:52