1
$\begingroup$

I'm fitting an SVC model with linear kernel, after that i'm checking the dot product of the fitted weights by the input to understand the prediction better. From my understanding of the linear SVC model, $x_1$ will be assigned to the positive class if

$$ w \cdot x_1 + b > 1 $$

and the negative class if

$$ w \cdot x_1 + b < -1 $$

However calculating the dot product by hand is not giving any clear cut as you can see below. enter image description here

Is my understanding of how SVC generates the prediction after learning the weights of the features correct or am i missing something?

It's worth mentioning that the values in the histogram are identical to what i get from decision_function(X) so to rephrase my question: how did the estimator in scikit-learn decide what values of the decision function are assigned to what class? Since as you can see from the histogram decision_function > 0 are not assigned to the positive class and decisioN_function < 0 are not assigned to the negative class.

$\endgroup$
2
$\begingroup$

The conditions you quote are for training: SVM attempts to find $w$ and $b$ such that

$$ y_i (w \cdot x_i + b) \ge 1 $$

where $y_i \in \{-1, 1\}$.

However, this can only succeed if the classes are linearly separable. If not, as your case suggests, you need a soft margin SVM, by introducing slack variables:

$$ y_i (w \cdot x_i + b) \ge 1 - \xi_i $$

For classification, the result is simply compared to 0: If $w \cdot x_i + b > 0$, classify $x_i$ to class "$+1$", and otherwise to "$-1$".

$\endgroup$
4
  • $\begingroup$ But even if it's compared to 0, the dot product of w and xi is not greater than 0 for the predicted instances of the positive class and similarly it's not below zero for the negative class $\endgroup$ Feb 9 at 7:21
  • $\begingroup$ I don't understand. What is the dot product then? Is it zero for all $x_i$, or does it consistently have the wrong sign (i.e. the classifier consistently misclassifies the data)? Can you provide some examples? $\endgroup$
    – Igor F.
    Feb 9 at 7:47
  • $\begingroup$ The dot product is identical to what i get from decision_function. It doesn't have the wrong sign but there's no clear cut where inputs with decision_function > 0 are assigned the positive class and the inputs with decision_function < 0 are assigned negative class $\endgroup$ Feb 9 at 7:49
  • $\begingroup$ Can you please post the code with example data? $\endgroup$
    – Igor F.
    Feb 9 at 8:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.