1
$\begingroup$

So I have a pretty well testing SVC train series which puts me into the mid 80 percentile without outrageous C/g values. My current C value is 2.0 and gamma is 0.5. Good numbers across the range during refinement - looking solid. Here's the cross-validation plot from my grid search:

enter image description here

I have been working on the libsvm command line as well as am writing C# test code via libsvm.net. On both sides I am experiencing very strange behavior. On the command line it happens when I change the default labels of the test series. On the C# side I am not supplying any labels, which may be incorrect, I don't see good examples that separate the training from the test data. In any case those test series labels should be ignored, right?

So this is a subset of my test series - the first ten rows. Let's call that test0:

1 1:-0.2 2:1 3:-1 4:-1 5:0.2 6:1 7:-0.6 8:-0.6 9:0.2 10:-0.6 11:-0.6 12:-0.6
0 1:1 2:0.2 3:-0.6 4:-0.6 5:1 6:0.2 7:0.2 8:-1 9:1 10:-0.6 11:0.6 12:-0.6
1 1:0.2 2:-1 3:0.2 4:0.6 5:0.2 6:-1 7:0.2 8:1 9:0.2 10:-1 11:-0.2 12:1
1 1:-0.2 2:-1 3:0.2 4:1 5:-0.2 6:-1 7:0.6 8:1 9:-0.6 10:-1 11:0.6 12:0.6
1 1:-0.6 2:-0.2 3:-0.6 4:0.6 5:-1 6:0.6 7:1 8:0.2 9:-1 10:0.6 11:0.6 12:-0.2
1 1:1 2:-0.2 3:-1 4:-1 5:1 6:0.2 7:-1 8:-0.6 9:1 10:-0.6 11:-0.6 12:-0.2
1 1:1 2:-0.2 3:-1 4:0.6 5:1 6:-0.2 7:0.6 8:0.2 9:0.6 10:1 11:-0.2 12:-1
1 1:-0.2 2:-0.6 3:-0.6 4:1 5:-0.2 6:-0.6 7:0.6 8:1 9:-0.6 10:-1 11:0.6 12:1
-1 1:0.6 2:1 3:-0.6 4:-1 5:0.6 6:0.6 7:-0.6 8:-1 9:0.6 10:-0.2 11:-0.6 12:-0.6
0 1:1 2:-0.6 3:-0.6 4:1 5:0.2 6:-0.6 7:1 8:0.2 9:-0.6 10:0.6 11:1 12:-0.6

I run that against my model and this is what I'm getting in my predict0 file:

1
0
1
1
1
1
1
1
-1
1

The command line shows: Accuracy = 90% (9/10) (classification)

Excellent - this is what we want to see. But obviously on my C# end I'm not supplying any labels. Which is why I'm seeing different results there. In order to double check this on the LIBSVM command line I changed all the labels to 0 - here now is my test1 file:

0 1:-0.2 2:1 3:-1 4:-1 5:0.2 6:1 7:-0.6 8:-0.6 9:0.2 10:-0.6 11:-0.6 12:-0.6
0 1:1 2:0.2 3:-0.6 4:-0.6 5:1 6:0.2 7:0.2 8:-1 9:1 10:-0.6 11:0.6 12:-0.6
0 1:0.2 2:-1 3:0.2 4:0.6 5:0.2 6:-1 7:0.2 8:1 9:0.2 10:-1 11:-0.2 12:1
0 1:-0.2 2:-1 3:0.2 4:1 5:-0.2 6:-1 7:0.6 8:1 9:-0.6 10:-1 11:0.6 12:0.6
0 1:-0.6 2:-0.2 3:-0.6 4:0.6 5:-1 6:0.6 7:1 8:0.2 9:-1 10:0.6 11:0.6 12:-0.2
0 1:1 2:-0.2 3:-1 4:-1 5:1 6:0.2 7:-1 8:-0.6 9:1 10:-0.6 11:-0.6 12:-0.2
0 1:1 2:-0.2 3:-1 4:0.6 5:1 6:-0.2 7:0.6 8:0.2 9:0.6 10:1 11:-0.2 12:-1
0 1:-0.2 2:-0.6 3:-0.6 4:1 5:-0.2 6:-0.6 7:0.6 8:1 9:-0.6 10:-1 11:0.6 12:1
0 1:0.6 2:1 3:-0.6 4:-1 5:0.6 6:0.6 7:-0.6 8:-1 9:0.6 10:-0.2 11:-0.6 12:-0.6
0 1:1 2:-0.6 3:-0.6 4:1 5:0.2 6:-0.6 7:1 8:0.2 9:-0.6 10:0.6 11:1 12:-0.6

And here's the predict1 file:

1
0
1
1
1
1
1
1
-1
1

Same predictions - very nice. However the command line gives me this:

Accuracy = 10% (1/10) (classification)

Say again? The predict2 file is correct 9 out of 10 times. That would be a minor hick-up - however on the C# side I'm getting incorrect predictions 50% of the times. I double checked the vectors that go into svm.Predict() and they are identical. The XML model on that end produces the identical output that I see on the command line version of LIBSVM, so I'm sure it's loading the right train file and gets the same settings.

I also tried other faux labels - one with all 3s - per the above my categories only allow -1, 0, and 1. Same results and same screwy output from LIBSVM.

Here's the method I wrote in C# - it's extremely simple:

/// <summary>
/// Makes the prediction based on the supplied data vector.
/// </summary>
/// <param name="x">the vector</param>
/// <returns>the category prediction as a double</returns>
public double Predict(double[] vector)
{
   svm_node[] x = new svm_node[vector.Length];
   for(int j = 0 ; j < vector.Length ; j++) // Save values for each attributes
    {
        x[j] = new svm_node() { index = j, value = vector[j] };
    }

    double predict = cSvm.Predict(x);
    return predict;
}

That's it - exceedingly simple. I'm producing a single vector, which I'm feeding into libsvm. Perhaps I'm doing something wrong here as I'm getting the following output on that end:

1:-0.2 2:1 3:-1 4:-1 5:0.2 6:1 7:-0.6 8:-0.6 9:0.2 10:-0.6 11:-0.6 12:-0.6
Expected: 1 - prediction: 1 - correct.
1:1 2:0.2 3:-0.6 4:-0.6 5:1 6:0.2 7:0.2 8:-1 9:1 10:-0.6 11:0.6 12:-0.6
Expected: 0 - prediction: 1 - incorrect.
1:0.2 2:-1 3:0.2 4:0.6 5:0.2 6:-1 7:0.2 8:1 9:0.2 10:-1 11:-0.2 12:1
Expected: 1 - prediction: 0 - incorrect.
1:-0.2 2:-1 3:0.2 4:1 5:-0.2 6:-1 7:0.6 8:1 9:-0.6 10:-1 11:0.6 12:0.6
Expected: 1 - prediction: -1 - incorrect.
1:-0.6 2:-0.2 3:-0.6 4:0.6 5:-1 6:0.6 7:1 8:0.2 9:-1 10:0.6 11:0.6 12:-0.2
Expected: 1 - prediction: 0 - incorrect.
1:1 2:-0.2 3:-1 4:-1 5:1 6:0.2 7:-1 8:-0.6 9:1 10:-0.6 11:-0.6 12:-0.2
Expected: 1 - prediction: 1 - correct.
1:1 2:-0.2 3:-1 4:0.6 5:1 6:-0.2 7:0.6 8:0.2 9:0.6 10:1 11:-0.2 12:-1
Expected: 1 - prediction: 1 - correct.
1:-0.2 2:-0.6 3:-0.6 4:1 5:-0.2 6:-0.6 7:0.6 8:1 9:-0.6 10:-1 11:0.6 12:1
Expected: 1 - prediction: -1 - incorrect.
1:0.6 2:1 3:-0.6 4:-1 5:0.6 6:0.6 7:-0.6 8:-1 9:0.6 10:-0.2 11:-0.6 12:-0.6
Expected: -1 - prediction: 1 - incorrect.
1:1 2:-0.6 3:-0.6 4:1 5:0.2 6:-0.6 7:1 8:0.2 9:-0.6 10:0.6 11:1 12:-0.6
Expected: 0 - prediction: -1 - incorrect.

As you can see seven out of ten are incorrect here. Same input data. I'm really scratching my head here. Here is the output I get during model creation from my C# test:

......*
optimization finished, #iter = 6318
nu = 0.29823202442349317
obj = -2600.72689660147, rho = 0.1313953415634528
nSV = 2067, nBSV = 1324
.....*
optimization finished, #iter = 5661
nu = 0.3084318050488557
obj = -2492.6705073363432, rho = 0.10201329253787896
nSV = 1938, nBSV = 1290
.....*
optimization finished, #iter = 5079
nu = 0.0938971979545962
obj = -805.0415458108344, rho = 0.004972540301229775
nSV = 1095, nBSV = 400
Total nSV = 3700

And here's the output LIBSVM gives me on the command shell using the svm-train command:

....*..*
optimization finished, #iter = 6318
nu = 0.298232
obj = -2600.726897, rho = 0.131395
nSV = 2067, nBSV = 1324
...*..*
optimization finished, #iter = 5661
nu = 0.308432
obj = -2492.670507, rho = 0.102013
nSV = 1938, nBSV = 1290
...*..*
optimization finished, #iter = 5079
nu = 0.093897
obj = -805.041546, rho = 0.004973
nSV = 1095, nBSV = 400
Total nSV = 3700

Both are identical so I am certainly producing the same model with my C# code. Am I perhaps misunderstanding the API? In any case the command line behavior (i.e. incorrect reporting of test results) is strange as well although I do appreciate getting solid output on that side.

Any help/input/insights/suggestions would be very welcome. Thanks in advance.

$\endgroup$
1
$\begingroup$

I hope answering one's own question is permissible here (such strict rules!):

Anyway, I figured out what was going on after I took a closer look at the vector that was produced. Turns out I was mislabeling the svm-nodes in my vector. The correct code is below:

/// <summary>
/// Makes the prediction based on the supplied data vector.
/// </summary>
/// <param name="x">the vector</param>
/// <returns>the category prediction as a double</returns>
public double Predict(double[] vector)
{
    svm_node[] x = new svm_node[vector.Length];
    for(int j = 0 ; j < vector.Length ; j++) // Save values for each attributes
    {
        x[j] = new svm_node() { index = j + 1, value = vector[j] }; // changed to j + i
    }

    double predict = cSvm.Predict(x);
    return predict;
}

The loop is zero based and of course LIBSVM expects the labels to start at 1. I hope the corrected code helps someone else.

$\endgroup$
  • $\begingroup$ Awesome--glad you figured it out! You can even mark your own answer as accepted (you don't get the bonus reputation though). Anyway, welcome to Cross Validated and hope we see you around! $\endgroup$ – Matt Krause Sep 22 '14 at 20:31
  • $\begingroup$ Thanks Matt - just like their vector format LIBSVM's documentation is sparse ;-) Any little bit helps... $\endgroup$ – Michael Sep 23 '14 at 7:15
0
$\begingroup$

I've never used the C# bindings, but the command-line part is trivial: you put garbage into the class labels, so libSVM output some garbage for the accuracy measurement.

This is a reasonable thing to do--you have to put something in that field, especially when you're in production and may not have ground-truth data. However it does mean that LibSVM cannot measure your model's accuracy (how could it?). You were just particularly unlucky that you chose "0" as the fill value: 1 of the 10 examples belongs to class 0, which gives you 10% accuracy. This unfortunately happens to be the duel of (100%-10%) of the model's actual accuracy too, even though they're unrelated.

$\endgroup$
  • 1
    $\begingroup$ Yes, I just realized that LIBSVM has no way of verifying the veracity of the answer unless I provide the proper labels in the test file. Silly me!! LOL :-) However, there's still something wrong with the way I'm creating the svm-node, but if you haven't used the C# (actually Java) API then you'll probably won't be able to help me. But thanks for your quick response - much obliged. $\endgroup$ – Michael Sep 20 '14 at 13:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.