Adding data to change SVM classification

Question

I have worked with SVMs in the past and I consider myself quite familiar with their main aspects. But an answer in here surprised me quite a bit.

The original poster asked for a simple case of separating 4 2-d points in 2 classes: points were: [ 0,0 ], [ 0,1 ], [ 1,0 ], [ 1,1 ] and labels were: [ -1, -1, -1, 1 ].

The problem was that with the default parameters and linear model it did not separate the data correctly.

A proposed method was to add more data to the dataset. The funny thing is that the data added was just an exact replicate of the original 4 points. 5 times to be precise. So, the data became:

[ 0,0 ], [ 0,1 ], [ 1,0 ], [ 1,1 ], [ 0,0 ], [ 0,1 ], [ 1,0 ], [ 1,1 ], [ 0,0 ], [ 0,1 ], [ 1,0 ], [ 1,1 ], [ 0,0 ], [ 0,1 ], [ 1,0 ], [ 1,1 ], [ 0,0 ], [ 0,1 ], [ 1,0 ], [ 1,1 ]

with the corresponding labels also quintupled.

My question is whether should this work in SVM in general. I found it wrong (as a solution) as I consider adding data that does not provide new discrimination values could not affect the result. In this implementation it does affect it though. So, is this consistent with theory or is it an implementation issue? What am I missing here?

P.S.

The implementation is in python, sklearn.svm to be precise if it's of any concern.

Answer 1

This is related to the "Soft" margin solution. If you recall the equation from the optimization problem, you have:

$$min_{\textbf{w}\epsilon\mathbb{R}^{d},\xi_{i}\epsilon\mathbb{R}^{+}}\left |\right|\textbf{w}\left |\right|^{2}+C\sum_{i}^{N}\xi_i$$

subject to:

$$y_i(\textbf{w}^{T}\textbf{x}_{i}+b)\geq 1-\xi_{i} \quad \textrm{for} \quad i=1\ldots N$$

Here, we see that the constant C allows points outside the optimal margins to happen. The value of this constant is not dependent on other values, it's an absolute value. Therefore, if the value is small and you only have 1 fishy sample (as in the data of the example), the penalty is so small that the algorithm just don't care about one missed point.

By adding more points, we also increase the whole penalty term and the algorithm learns to avoid missclassifying those points (in a sense it's much more concerned about missed points). Try setting the value of C to something big (let's say 5, which is somehow equivalent to add 5 times more data) and you will see that it converges into the same coefficients of the example that uses the data times 5. Both approaches are simply increasing the whole penalty term, just in a different way.

from sklearn.svm import SVC

data = [[ 0,0 ], [ 0,1 ], [ 1,0 ], [ 1,1 ] ]
category = [ -1,  -1,  -1, 1 ]

#Original data, with C=1 (default value):
clf = SVC(kernel='linear', C = 1)
clf.fit(data, category)
print('coefs = '+str(clf.coef_[0])+', b = '+str(clf.intercept_[0])+', Score = '+str(clf.score(data, category)))

coefs = [ 0.     0.001], b = -1.0005, Score = 0.75 # Poorly classified

#Data times 5, same C:
clf = SVC(kernel='linear')
clf.fit(data*5, category*5)
print('coefs = '+str(clf.coef_[0])+', b = '+str(clf.intercept_[0])+', Score = '+str(clf.score(data, category)))

coefs = [ 2.  2.], b = -3.0, Score = 1.0 # Better results


#Original data, C=5:
clf = SVC(kernel='linear', C=5)
clf.fit(data, category)
print('coefs = '+str(clf.coef_[0])+', b = '+str(clf.intercept_[0])+', Score = '+str(clf.score(data, category)))

coefs = [ 2.  2.], b = -3.0, Score = 1.0 # Same results as when increasing the data

In my opinion, there is not a good practice just adding repeated data points just like that. There is always a better way of making your algorithm improve.

Hope this helps!