0
$\begingroup$

I was going through this scikit-learn link and I noticed, OneClassSVM is trained on normal and outlier both. Specifically, they are adding the outliers in the following line:

X = np.concatenate([X, rng.uniform(low=-6, high=6, size=(n_outliers, 2))], axis=0)

and training and predicting:

 y_pred = algorithm.fit_predict(X)

But, what I understood about one class SVM is that, first we have to train it with normal data so that it learns "what is normal", then if during predicting, if we give outlier it would give a different score (based on the distance from the support vector ) than it would have given if we would feed it with normal examples.

but I see a different case in that link. and that actually gave good results too.

so this is what they did basically:

from sklearn.datasets import make_blobs, make_moons
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
from sklearn.svm import OneClassSVM

cluster=make_blobs(centers=[[0, 0], [0, 0]], cluster_std=0.5,random_state=0, n_samples=50, n_features=2)[0]

noise=rng.uniform(low=-6, high=6, size=(n_outliers, 2))

cluster_and_noise=np.concatenate([cluster, noise], axis=0)

##use all the data to fit
clf = OneClassSVM(gamma='auto').fit(cluster_and_noise)
y_pred=clf.predict(cluster_and_noise)

ax1 = plt.subplot(1, 3, 1)
ax1.set_xlim(left=-7, right=7)
ax1.set_ylim(bottom=-7, top=7)
ax1.scatter(cluster[:,0],cluster[:,1])

ax2 = plt.subplot(1, 3, 2, sharey=ax1, sharex=ax1)
ax2.scatter(noise[:,0],noise[:,1])

ax3 = plt.subplot(1, 3, 3, sharey=ax1, sharex=ax1)
ax3.scatter(cluster_and_noise[:,0],cluster_and_noise[:,1],color=colors[(y_pred + 1) // 2])

enter image description here

but then if use the same classifier and get scores, we see the scores of cluster and noise is not that different.

clf.score_samples(cluster)
>>. array([4.00469685, 3.83800399, 3.96132707, 3.9558005 , 3.95252136,
       3.979525  , 4.0142477 , 3.83803278, 3.99085042, 4.001514  ,
       3.95015783, 3.96439505, 3.90314513, 3.96228951, 3.98460255,
       3.93268752, 3.94045451, 3.89491044, 3.9889579 , 3.94145627,
       3.94766615, 3.84360208, 3.95178757, 3.98853109, 3.95894264,
       4.00293265, 4.01293144, 3.83284988, 3.84374284, 3.9944587 ,
       3.96554754, 3.93684397, 4.00709846, 3.95332379, 3.77089662,
       3.95696066, 3.84374744, 3.95638626, 4.01814036, 4.03626033,
       3.95493713, 3.9958754 , 3.948184  , 4.01123206, 3.98196151,
       3.88720297, 3.84337515, 4.01407151, 4.0429624 , 3.98858635])

clf.score_samples(noise)
>> array([1.37120633, 2.33386487, 3.84350039, 2.54988743, 3.84353343,
       2.91550353, 3.92382769, 2.51407581, 3.77670169, 1.55465272,
       2.96472539, 3.56875574, 3.30662736, 3.50916248, 2.11246036,
       3.83174597, 3.50537602, 1.97770223, 1.15322786, 2.59337546,
       1.33454276, 3.49754735, 3.55715574, 3.29734357, 3.84322719,
       2.64399899, 3.54421041, 2.28127403, 3.16844724, 3.5192065 ,
       3.84350038, 3.8433305 , 1.93984445, 3.17917933, 3.84354178,
       1.44641648, 1.88731014, 2.01971328, 3.27891961, 1.44334252,
       3.39863946, 3.56970341, 3.83036641, 3.72702916, 2.95517646])

it is hard to have a threshold and predict based on the side of the threshold.

but if we train on only the normal data and get scores, then the scores are really different:

clf1 = OneClassSVM(gamma='auto').fit(cluster)
clf1.score_samples(cluster)
array([15.53623443, 12.08894094, 13.30125568, 15.49971357, 13.99998325,
       16.31439861, 15.85847211, 10.75065984, 14.80507958, 15.15117585,
       15.33981291, 16.24634777, 11.63374312, 15.95374769, 16.32038695,
       14.09045035, 14.10018567, 12.3558028 , 16.39410368, 13.90079851,
       13.25848217, 13.03936213, 14.80428242, 14.36238355, 15.63797517,
       15.45050203, 15.90881324,  9.27659238,  9.69184167, 16.33307024,
       16.26757746, 14.14340524, 15.35303742, 15.35834253, 11.01129106,
       14.80507939, 10.8731272 , 15.55793974, 14.73577099, 13.05983176,
       13.52727811, 16.34075366, 13.21925331, 16.18221564, 16.2906526 ,
       11.78899607, 10.1191241 , 15.80523227, 13.01352676, 16.03966244])
clf1.score_samples(noise)
array([ 0.0000885 ,  0.10726473,  3.20817757,  0.00166576,  0.42899449,
        0.02670632, 12.8523114 ,  0.00638219,  0.00000002,  0.32227899,
        0.02508491,  0.00000001,  4.27744142,  7.57454283,  0.00096782,
        0.30527275,  5.33172462,  0.00000019,  0.00000087,  0.76838717,
        0.00004217,  3.68289332,  0.19237365,  0.00000013,  0.00000511,
        0.00806763,  0.01658629,  0.0035727 ,  0.01107006,  0.00000038,
        0.0021924 ,  0.00000256,  0.00696933,  0.05531983,  3.13177134,
        0.00000134,  0.00032246,  0.0000005 ,  0.00001288,  0.005572  ,
        0.00051327,  7.35481314,  3.73427369,  0.77003368,  0.00000157])

So we can easily differentiate the noise from the normal cluster from from this score.

Now, after this experiment, I am confused. because the scores tell me i am right about my understanding of OneClassSVM, but then why does the prediction function work when the model trained all the data? I thought predict() uses the score and has a default score or something.

How is that predict() function working? Since we are not giving any labels, how does the model knows the 2 class? using density? but I thought SVM does not do any density approximation.

$\endgroup$

1 Answer 1

0
$\begingroup$

One-class training for anomaly detection is supposed to only get "normal" class data as input. However, it is very common in practice that what we believe to be "normal" data is in fact contaminated by some outliers. Because of this, it is highly desirable that the AD method can reject such contamination, and find the true normal datapoints.

For one-class SVM the ability to reject outliers is highly dependent on the regularization parameters. In scikit-learn OneClassSVM the most relevant parameters should be gamma and nu.

The one-class SVM learns a hyperplane, and this separates the one normal class from all other classes. The method decision_function() can be used to get the (signed) distance to this hyperplane, it returns positive for an inlier and negative for an outlier.

The predict function uses the continious scores, but also applies a learned _offset in order to binarize into inlier/outlier.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.