# Why can't scikit-learn SVM solve two concentric circles?

Consider the following dataset (code for generating it is at the bottom of the post): Running the following code:

from sklearn.svm import SVC
model_2 = SVC(kernel='rbf', degree=2, gamma='auto', C=100)
model_2.fit(X_train, y_train)
print('accuracy (train): %5.2f'%(metric(y_train, model_2.predict(X_train))))
print('accuracy (test): %5.2f'%(metric(y_test, model_2.predict(X_test))))
print('Number of support vectors:', sum(model_2.n_support_))


I get the following output:

accuracy (train):  0.64
accuracy (test):  0.26
Number of support vectors: 55


I also tried with varying degrees of polynomial kernel and got more or less the same results.

So why does it do such a poor job. I've just learned about SVM and I would have thought that a polynomial kernel of 2nd degree could just project these points onto a paraboloid and the result would be linearly separable. Where am I going wrong here?

Reference: The starter code for the snippets in this post comes from this course

Code for generating data:

np.random.seed(0)
data, labels = sklearn.datasets.make_circles()
idx = np.arange(len(labels))
np.random.shuffle(idx)
# train on a random 2/3 and test on the remaining 1/3
idx_train = idx[:2*len(idx)//3]
idx_test = idx[2*len(idx)//3:]
X_train = data[idx_train]
X_test = data[idx_test]

y_train = 2 * labels[idx_train] - 1  # binary -> spin
y_test = 2 * labels[idx_test] - 1

scaler = sklearn.preprocessing.StandardScaler()
normalizer = sklearn.preprocessing.Normalizer()

X_train = scaler.fit_transform(X_train)
X_train = normalizer.fit_transform(X_train)

X_test = scaler.fit_transform(X_test)
X_test = normalizer.fit_transform(X_test)
plt.figure(figsize=(6, 6))
plt.subplot(111)
plt.scatter(data[labels == 0, 0], data[labels == 0, 1], color='navy')
plt.scatter(data[labels == 1, 0], data[labels == 1, 1], color='c')
$$$$

• Thanks for looking into it. The original code used the default value of C (didn't specify it). That was me just playing around with it to try and illicit different results. – Alexander Soare Jun 17 '20 at 12:19

1. All the preprocessing should be done using training set's fitted values:

X_test = scaler.transform(X_test)
X_test = normalizer.transform(X_test)

2. degree is a hyperparameter for polynomial kernel and is ignored if the kernel is not poly:

model_2 = SVC(kernel='poly', degree=2, gamma='auto', C=100)


OR

model_2 = SVC(kernel='rbf', gamma='auto', C=100)

3. While debugging, print the final dataset after going through preprocessing to see if you've destroyed the dataset: Do not blindly implement preprocessing. Remove the normalisation step because it just sabotages the dataset. You'll have 100% accuracy.

• I guess another tip is to not blindly follow online course code even if it seems to come from an "official" source... Thanks! – Alexander Soare Jun 17 '20 at 12:23
• +1 for detailed and fast answer! – Haitao Du Jun 17 '20 at 12:25
• very interesting to see we cannot blindly apply pre-processing. i think many people things pre-processing and scale data is always better... – Haitao Du Jun 17 '20 at 12:33
• yeah, almost nothing is always better. In this specific toy example, it is the scale that differentiates the classes. – gunes Jun 17 '20 at 12:35
• (+1): Might help to explain what normalizer.fit_transform does: "Each sample (i.e. each row of the data matrix) with at least one non zero component is rescaled independently of other samples so that its norm (l1, l2 or inf) equals one". The L2-norm is the default, so it precisely places each pair of observations on the unit circle. – Scortchi - Reinstate Monica Jun 17 '20 at 21:19

@gunes has a very good answer: degree is for poly, and rbf is controlled by gamma and C. In general, it is not surprising to see the default parameter does not work well.

model_2 = SVC(kernel='rbf', gamma=1000, C=100)`