0
$\begingroup$

I'm not certain I understand how sklearn's Linear SVC works. I had assumed that it would find an optimal hyper-plane to divide one class from another.

I tried to recover the separating hyper-plane from the following example in the docs (https://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html#sklearn.svm.LinearSVC.score):

from sklearn.svm import LinearSVC
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import make_classification
X, y = make_classification(n_features=4, random_state=0)

clf = make_pipeline(StandardScaler(), LinearSVC(random_state=0, tol=1e-5))
clf.fit(X, y)
clf.score(X, y)

and, found the score was 0.93

I then tried to extract the dividing hyper-plane and recover this result. I did it the following way:

A = clf.named_steps['linearsvc'].coef_
b = clf.named_steps['linearsvc'].intercept_
C = np.dot(A,X.transpose()) + b
C = C[0,:] # This gives C the shape of y

I then tried to rescale C so that all positive responses correspond to +1 and all negative responses correspond to 0. This was to make C match y. I did this as follows, renaming C as y_calc:

y_calc = (C > 0).astype(np.int)
clf.score(X, y_calc)

I expected to get a perfect score by using y_calc as the 2nd argument to clf.score. I did not. The score I got was 0.96.

This doesn't make sense. I eventually found that if I added a margin, I could get the perfect score I expected:

C1 = C + 2.5e-1
y_calc = (C1 > 0).astype(np.int)
clf.score(X, y_calc)

I don't understand this. I found 2.5e-1 by guessing. I don't see where it came from. The specified tolerance in LinearSVC was 1e-5, not 2.5e-1.

Why did I need to add 2.5e-1 to C, in order to get the classifications from clf.score mach those calculated directly from the SVC calculated hyper-plane?


I think the issue lies in the use of StandardScaler() within make_pipeline. Possibly, I need to extract the standard scaling.

$\endgroup$
0
$\begingroup$

Yes .... The issue was standard scaling. I needed to rescale X, by subtracting the component-wise mean (mu=np.mean(X, axis=0)) and dividing by the component-wise std (sigma = np.sqrt(np.var(X, axis=0)) to recover the correct result.

It should be noted that in their code the authors of sklearn.svm are careful to avoid divide by zero errors, deal with issues involving sparse/non-sparse matrices and update incremental statistics using the following reference:

"The algorithm for incremental mean and std is given in Equation 1.5a,b
        in Chan, Tony F., Gene H. Golub, and Randall J. LeVeque. "Algorithms
        for computing the sample variance: Analysis and recommendations."
        The American Statistician 37.3 (1983): 242-247:"
    

I did not need to be so careful for my quick sanity check. Using the following rescaling was all that was needed to correctly use the Linear SVM's hyper-plane:

mu = np.mean(X, axis=0)
sigma_2 = np.var(X, axis=0)
sigma = np.sqrt(sigma_2) #Slightly too slavish in mimicing sklearn's calculations
                         # Could've just calc'ed np.std directly
Xscale = (X-mu)/sigma
Cscale = np.dot(A,Xscale.transpose()) + b
Cscale = Cscale[0,:]
y_calc = (Cscale  > 0).astype(np.int)
clf.score(X,y_calc)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.