13
$\begingroup$

Consider the following dataset (code for generating it is at the bottom of the post): enter image description here

Running the following code:

from sklearn.svm import SVC
model_2 = SVC(kernel='rbf', degree=2, gamma='auto', C=100)
model_2.fit(X_train, y_train)
print('accuracy (train): %5.2f'%(metric(y_train, model_2.predict(X_train))))
print('accuracy (test): %5.2f'%(metric(y_test, model_2.predict(X_test))))
print('Number of support vectors:', sum(model_2.n_support_))

I get the following output:

accuracy (train):  0.64
accuracy (test):  0.26
Number of support vectors: 55

I also tried with varying degrees of polynomial kernel and got more or less the same results.

So why does it do such a poor job. I've just learned about SVM and I would have thought that a polynomial kernel of 2nd degree could just project these points onto a paraboloid and the result would be linearly separable. Where am I going wrong here?

Reference: The starter code for the snippets in this post comes from this course

Code for generating data:

np.random.seed(0)
data, labels = sklearn.datasets.make_circles()
idx = np.arange(len(labels))
np.random.shuffle(idx)
# train on a random 2/3 and test on the remaining 1/3
idx_train = idx[:2*len(idx)//3]
idx_test = idx[2*len(idx)//3:]
X_train = data[idx_train]
X_test = data[idx_test]

y_train = 2 * labels[idx_train] - 1  # binary -> spin
y_test = 2 * labels[idx_test] - 1

scaler = sklearn.preprocessing.StandardScaler()
normalizer = sklearn.preprocessing.Normalizer()

X_train = scaler.fit_transform(X_train)
X_train = normalizer.fit_transform(X_train)

X_test = scaler.fit_transform(X_test)
X_test = normalizer.fit_transform(X_test)
plt.figure(figsize=(6, 6))
plt.subplot(111)
plt.scatter(data[labels == 0, 0], data[labels == 0, 1], color='navy')
plt.scatter(data[labels == 1, 0], data[labels == 1, 1], color='c')
```
$\endgroup$
1
  • $\begingroup$ Thanks for looking into it. The original code used the default value of C (didn't specify it). That was me just playing around with it to try and illicit different results. $\endgroup$ Jun 17 '20 at 12:19
18
$\begingroup$

Let's start with warnings:

  1. All the preprocessing should be done using training set's fitted values:

    X_test = scaler.transform(X_test)
    X_test = normalizer.transform(X_test)
    
  2. degree is a hyperparameter for polynomial kernel and is ignored if the kernel is not poly:

    model_2 = SVC(kernel='poly', degree=2, gamma='auto', C=100)
    

    OR

    model_2 = SVC(kernel='rbf', gamma='auto', C=100)
    
  3. While debugging, print the final dataset after going through preprocessing to see if you've destroyed the dataset:

enter image description here

Do not blindly implement preprocessing. Remove the normalisation step because it just sabotages the dataset. You'll have 100% accuracy.

$\endgroup$
6
  • 1
    $\begingroup$ I guess another tip is to not blindly follow online course code even if it seems to come from an "official" source... Thanks! $\endgroup$ Jun 17 '20 at 12:23
  • 3
    $\begingroup$ +1 for detailed and fast answer! $\endgroup$
    – Haitao Du
    Jun 17 '20 at 12:25
  • 2
    $\begingroup$ very interesting to see we cannot blindly apply pre-processing. i think many people things pre-processing and scale data is always better... $\endgroup$
    – Haitao Du
    Jun 17 '20 at 12:33
  • 2
    $\begingroup$ yeah, almost nothing is always better. In this specific toy example, it is the scale that differentiates the classes. $\endgroup$
    – gunes
    Jun 17 '20 at 12:35
  • 4
    $\begingroup$ (+1): Might help to explain what normalizer.fit_transform does: "Each sample (i.e. each row of the data matrix) with at least one non zero component is rescaled independently of other samples so that its norm (l1, l2 or inf) equals one". The L2-norm is the default, so it precisely places each pair of observations on the unit circle. $\endgroup$ Jun 17 '20 at 21:19
6
$\begingroup$

@gunes has a very good answer: degree is for poly, and rbf is controlled by gamma and C. In general, it is not surprising to see the default parameter does not work well.

See RBF SVM parameters


If you change your code

model_2 = SVC(kernel='rbf', gamma=1000, C=100)

You will see 100% on training but 56% on testing.

The reason is As @gunes mentioned the pre-processing changed the data. this also tells us RBF kernel is pretty powerful that can overfit training data pretty well.

$\endgroup$
1
  • $\begingroup$ (+1) Yeah, I've never mentioned the hyper-parameter choice part since it was pretty specified in the question. $\endgroup$
    – gunes
    Jun 17 '20 at 12:30
-3
$\begingroup$

The answer is very simple and very short. Because you attempt to make a support vector machine create something that is impossible, there is no support vectors that will constrain to only those two circles.

$\endgroup$
1
  • 4
    $\begingroup$ That's true for a linear SVM, but this should be a cakewalk when using a radial (i.e., circular) basis function--there's very clearly a circle that separates the two classes! $\endgroup$ Jun 18 '20 at 0:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.