3
$\begingroup$

I used the MATLAB interface of libsvm for doing binary classification of 997-dimensional training data. I am trying to understand how the resulting model is used to compute the predicted output (which we get by calling svmpredict)

The model contains fields (it has linear kernel):

nSV = [546; 246]; totalSV=792; rho = 0.093
and svCoeff [792x1 double] and SVs [792x997 double]

I thought that we must be simply multiplying svCoeff with SVs to get a [997x1] matrix which we then multiply with the actual feature, before shareholding by rho. But that's not the case. Can someone illustrate with a simple equation how these parameters are used to do classification?

$\endgroup$
6
$\begingroup$

Support vector machine classifiers use the following decision function to determine the label for a test instance $\mathbf{z}$:

$f(\mathbf{z})=\mathtt{sign}\big(\sum_{i=1}^{totalSV} y_i \alpha_i \kappa(\mathbf{x}_i,\mathbf{z})-\rho\big)=\mathtt{sign}\big(\langle\mathbf{w},\Phi(\mathbf{z})\rangle-\rho\big)$,

where $\kappa(\cdot,\cdot)$ is the kernel function, $\alpha$ contains the support values, $\mathbf{y}$ is the training label vector, $\rho$ is a bias term and $\mathbf{w}$ is the separating hyperplane in feature space.

In a LIBSVM model, sv_coef contains $\alpha_i y_i$ and SVs contains the support vectors ($\mathbf{x}_i$). To predict you need to perform kernel evaluations between the test point and all support vectors.

For the linear kernel ($\kappa(\mathbf{x},\mathbf{z})=\mathbf{x}^T\mathbf{z}$) you can compute $\mathbf{w}$ explicitly:

$\mathbf{w}=\sum_{i=1}^{totalSV} \alpha_i y_i \mathbf{x}_i=\mathtt{sv\_coef}^T \times \mathtt{SVs}$.

Subsequently, predictions are simply based on the sign of $\mathbf{w}^T\mathbf{z}-\rho$.

$\endgroup$
  • $\begingroup$ That's correct. But do you know why b=-rho? I haven't come across the rho notation in literature. It would be great if you could provide some reference using the -rho notation. thanks $\endgroup$ – Rishi Dua Oct 8 '14 at 5:22
  • 1
    $\begingroup$ Also w = SVs'*coef and not coef'*SVs. Please correct it $\endgroup$ – Rishi Dua Oct 8 '14 at 5:34
0
$\begingroup$

Nevermind, I found svm.cpp in the svmlight package and read the svm_predict function. It is written for the general case for n classes but for the simple case of two classes their logic boild down to

>> sv=model.SVs;
>> svc=model.sv_coef;
>> sv546=sv(1:546, :); %Since model.label is [1, -1] and model.nSV=[546; 246]
>> sv246=sv(547:end, :);
>> svc546=svc(1:546);
>> svc246=svc(547:end);
>> weight_for_minus1=transpose(svc246)*sv246; %Since model.label is [1, -1] and model.nSV=[546; 246]
>> weight_for_plus1=transpose(svc546)*sv546;
>> 'now multiply weight_for_minu1 and weight_for_plus1 with the 997-dimensional feature and select whichever is positive'
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.