12
$\begingroup$

I am trying to build an SVM from training data where one group is represented more than the other. However, the groups will be equally represented in the eventual test data. Therefore, I'd like to use the class.weights parameter of the e1071 R package interface to libsvm to balance the influence of the two groups in the training data.

Since I was unsure exactly how these weights should be specified, I set up a little test:

  1. Generate some null data (random features; 2:1 ratio between group labels)
  2. Fit an svm with the class.weights parameter set.
  3. Predict a bunch of new null datasets and look at the class proportions.
  4. Replicate the whole process many times for different null training sets.

Here is the R code I'm using:

nullSVM <- function(n.var, n.obs) {
    # Simulate null training data
    vars   = matrix(rnorm(n.var*n.obs), nrow=n.obs)
    labels = rep(c('a', 'a', 'b'), length.out=n.obs)
    data   = data.frame(group=labels, vars)

    # Fit SVM
    fit = svm(group ~ ., data=data, class.weights=c(a=0.5, b=1))

    # Calculate the average fraction of 'a' we would predict from null test data
    mean(replicate(50, table(predict(fit, data.frame(matrix(rnorm(n.var*n.obs), nrow=n.obs))))[1])) / n.obs
}

library(e1071)
set.seed(12345)
mean(replicate(50, nullSVM(50, 300)))

From this whole thing I was expecting an output ~ 0.5, however, that's not what I got:

> mean(replicate(50, nullSVM(50, 300)))
[1] 0.6429987

The class.weights paramter is working, sort of, as the lower I weight a, the lower it is represented in this simulation (and if I omit class.weights it returns close to 1)...but I do not understand why simply using weights of 1:2 (for training data that is 2:1) does not get me all the way down to 50%.

If I'm misunderstanding SVMs, can someone explain this point? (or send some refs?)

If I'm doing it wrong, can someone tell me the correct way to use the class.weights parameter?

Could it possibly be a bug? (I think not, since I understand this software and the underlying libsvm to be quite mature)

$\endgroup$
  • $\begingroup$ I don't have experience with libsvm but with LiblineaR, the class weights are crucial. Withouth setting it correctly, you get sub-optimal results if your classes are heavily unbalanced. I would suggest: Get a real dataset with unbalanced classes and try different values of class.weights (in LiblineaR wi). LiblineaR is orders of magnitude faster for a lineal kernel and has penalized methods also. In my experience, you first find a decent class weight and then optimize C. $\endgroup$ – marbel Oct 12 '14 at 21:59
7
$\begingroup$

I think it may depend on the values of C and the number of patterns you have. The SVM tries to find the maximum margin discriminant, so if you have sparse data then it is possible that the SVM might find the hard-margin solution without any of the Lagrange multipliers reaching their upper bounds (in which case the ratio of penalties for each class is essntially irrelevant as the slack-valiables are small or zero. Try increasing the number of training patterns and see if that has an effect (as that makes it less likely that the hard-margin solution can be found within the box-constraints).

More importantly, the optimal values of C are data-dependent, you can't just set them to some pre-determined values, but instead optimise them by minimising the leave-one-out error or some generalisation bound. If you have imbalanced classes, you can fix the ratio of values for each class, and optimise the average penalty over all patterns.

$\endgroup$
  • $\begingroup$ This makes sense. When I reduce the number of features and increase the number of observations in this simulation, the output value moves closer to 0.5. However, it never quite gets there - even with 900 rows and only 1 column. $\endgroup$ – John Colby Nov 5 '11 at 17:06
  • $\begingroup$ Of course on real data I always use the caret package or the built-in tune() function for model parameter tuning, so I especially like your second idea for how to deal with this in practice by adjusting the resampling scheme to favor the minority class. Much appreciated. $\endgroup$ – John Colby Nov 5 '11 at 17:08
  • $\begingroup$ glad you suggestion was useful. There is a paper on setting the optimal ratio which might also be useful theoval.cmp.uea.ac.uk/publications/pdf/ijcnn2001.pdf However, the optimal theoretical correction isn't always optimal in practice, so the best results might actually be obtained by tuning the two separate C parameters without forcing a particular ratio, but weighting the patterns according to class when evaluating the leave-one-out model selection criterion. $\endgroup$ – Dikran Marsupial Nov 5 '11 at 17:43
  • 2
    $\begingroup$ I'd also add, these days I tend to use kernel ridge regression rather than SVMs as you don't have these sort of counter-intuitive problems due to the discontinuity in the derivative of the loss function. Quite often if you tune an L2 SVM properly, you end up with a very small value of C and all the data are SVs, at which point you have a KRR model anyway. The more I used them, the less useful I have found SVMs in practice, although the theoretical insights they have brought have been vital. $\endgroup$ – Dikran Marsupial Nov 5 '11 at 17:47
0
$\begingroup$

in training svm find support vectors to make a discriminative boundary and when there is enough support vectors for all classes data for doing so, it would be no problem. in the results accuracy of test set you should mind the equality amount of data for all classes in real world and for obtaining real results you shoud manipulate the data as well as it is properly considered into real situation.

$\endgroup$
  • $\begingroup$ This answer is rather unclear at the moment - I considered giving it a copy-edit but there are several places where I was unsure what you intended. Punctuation and grammmar are important for conveying meaning. $\endgroup$ – Silverfish Apr 17 '16 at 8:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.