# How can I extract the correct hyper-plane from sklearn.svm's LinearSVC

I'm not certain I understand how sklearn's Linear SVC works. I had assumed that it would find an optimal hyper-plane to divide one class from another.

I tried to recover the separating hyper-plane from the following example in the docs (https://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html#sklearn.svm.LinearSVC.score):

from sklearn.svm import LinearSVC
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import make_classification
X, y = make_classification(n_features=4, random_state=0)

clf = make_pipeline(StandardScaler(), LinearSVC(random_state=0, tol=1e-5))
clf.fit(X, y)
clf.score(X, y)


and, found the score was 0.93

I then tried to extract the dividing hyper-plane and recover this result. I did it the following way:

A = clf.named_steps['linearsvc'].coef_
b = clf.named_steps['linearsvc'].intercept_
C = np.dot(A,X.transpose()) + b
C = C[0,:] # This gives C the shape of y


I then tried to rescale C so that all positive responses correspond to +1 and all negative responses correspond to 0. This was to make C match y. I did this as follows, renaming C as y_calc:

y_calc = (C > 0).astype(np.int)
clf.score(X, y_calc)


I expected to get a perfect score by using y_calc as the 2nd argument to clf.score. I did not. The score I got was 0.96.

This doesn't make sense. I eventually found that if I added a margin, I could get the perfect score I expected:

C1 = C + 2.5e-1
y_calc = (C1 > 0).astype(np.int)
clf.score(X, y_calc)


I don't understand this. I found 2.5e-1 by guessing. I don't see where it came from. The specified tolerance in LinearSVC was 1e-5, not 2.5e-1.

Why did I need to add 2.5e-1 to C, in order to get the classifications from clf.score mach those calculated directly from the SVC calculated hyper-plane?

I think the issue lies in the use of StandardScaler() within make_pipeline. Possibly, I need to extract the standard scaling.

Yes .... The issue was standard scaling. I needed to rescale X, by subtracting the component-wise mean (mu=np.mean(X, axis=0)) and dividing by the component-wise std (sigma = np.sqrt(np.var(X, axis=0)) to recover the correct result.

It should be noted that in their code the authors of sklearn.svm are careful to avoid divide by zero errors, deal with issues involving sparse/non-sparse matrices and update incremental statistics using the following reference:

"The algorithm for incremental mean and std is given in Equation 1.5a,b
in Chan, Tony F., Gene H. Golub, and Randall J. LeVeque. "Algorithms
for computing the sample variance: Analysis and recommendations."
The American Statistician 37.3 (1983): 242-247:"



I did not need to be so careful for my quick sanity check. Using the following rescaling was all that was needed to correctly use the Linear SVM's hyper-plane:

mu = np.mean(X, axis=0)
sigma_2 = np.var(X, axis=0)
sigma = np.sqrt(sigma_2) #Slightly too slavish in mimicing sklearn's calculations
# Could've just calc'ed np.std directly
Xscale = (X-mu)/sigma
Cscale = np.dot(A,Xscale.transpose()) + b
Cscale = Cscale[0,:]
y_calc = (Cscale  > 0).astype(np.int)
clf.score(X,y_calc)