0
$\begingroup$

A quick recap for what I want to do, I want to determine if a text is written by the same author or not. Thus I use one-class classification.
In my training set (18 samples), it looks like this (for simplifying, I used x as data value):

1 1:x 2:x "until" 200:x
1 1:x 2:x "until" 200:x

In my testing set (3 samples), it looks like this (for simplifying, I used y as data value):

1 1:y 2:y "until" 200:y

For data preparation (training and testing set), I set upper and lower scaling limit to +-1

-l -1 -u 1

For training, I use svm_type is one class svm, kernel type is sigmoid. Yet the accuracy is 0%

optimization finished, #iter = 13
obj = 22.901769047004553, rho = 5.476401914859387
nSV = 11, nBSV = 6
Accuracy = 0.0% (0/21) (classification)

Can someone show me what I did wrong here?
Thank you very much.

$\endgroup$
  • 1
    $\begingroup$ Did you use the same scaling for both training and testing set (e.g. did you use the same coefficients for training and testing)? If not, that is probably what's going on here. Also, any reason you're using the sigmoid kernel? It's rarely used, usually RBF is a better choice. Did you tune hyperparameters ($\nu$ and kernel parameters for one-class SVM)? $\endgroup$ – Marc Claesen Mar 5 '14 at 10:08
  • $\begingroup$ Hi, thanks for your comment, I'm new to this one, thus it would be very useful for me if you can explain them in a more detailed way. E.g. I'm not pretty sure how to use the same coefficients for training and testing .... $\endgroup$ – Xitrum Mar 5 '14 at 10:18
3
$\begingroup$

As per the comments, I suspect an issue with scaling. Another possibility is poor choice of hyperparameters, in the case of one-class SVM these are $\nu$ and kernel parameters.

Scaling

When using SVM's it is appropriate to scale your data, which you have done. In scaling, however, it is important to use the same coefficients to scale both training and testing set. This is explained in this practical guide to SVM classification (see 2.2 Scaling).

If you use different coefficients, your training and test sets become incompatible. The smaller your sets are, the larger this incompatibility may be (you are quite prone to this).

I am going to guess you scaled like this:

svm-scale -l -1 -u 1 train.txt
svm-scale -l -1 -u 1 test.txt

This is wrong! The scaling tool in LIBSVM internally computes coefficients based on the minimum and maximum per feature. Clearly these may differ between data sets (the larger, the less likely the difference will be substantial).

To ensure you use a single set of coefficients, use the following commands:

svm-scale -l -1 -u 1 -s coefs.txt train.txt
svm-scale -r coefs.txt test.txt

This saves the coefficients computed based on the training set and reuses them to scale the test set. This way they are compatible.

Hyperparameters and choice of kernel

When using SVM (any formulation), it is important to use optimal values of the hyperparameters. You used the sigmoid kernel (why?), which has the following kernel function: $$\kappa(\mathbf{x}_i,\mathbf{x}_j) = \tanh(\gamma \mathbf{x}_i^T\mathbf{x}_j + c_0)^{d}$$

This is quite a complex kernel functions with 3 tuning parameters (that is a lot). It is known to cause numerical issues. I suggest considering the RBF kernel instead, which has one tuning parameter and no numerical problems.

Since you have used one-class SVM with a sigmoid kernel, you have 4 parameters ($\nu$, $\gamma$, $c_0$ and $d$). Tuning all of these is going to be a hassle and will most definitely cause an overfit because your data sets are tiny. Yet another reason to get rid of the sigmoid kernel.

$\endgroup$
  • $\begingroup$ For the scaling part, I already got it correct. Then basing on your suggestion, I change to use RBF kernel. But the accuracy is still 0%.... Should it relate to (C,y) pair and something called -v n cross validation folds? $\endgroup$ – Xitrum Mar 5 '14 at 11:31
  • $\begingroup$ Yes, you have to do parameter selection. Typically this is done using cross-validation. $\endgroup$ – Marc Claesen Mar 5 '14 at 11:31
  • $\begingroup$ will be a stupid question... but how can I do it ? I looked up even in the document for beginner but could't find it .. $\endgroup$ – Xitrum Mar 5 '14 at 11:38
  • 1
    $\begingroup$ Searching for svm parameter selection on this site should help you get started. $\endgroup$ – Marc Claesen Mar 5 '14 at 11:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.