# SVM with unequal group sizes in training data

I am trying to build an SVM from training data where one group is represented more than the other. However, the groups will be equally represented in the eventual test data. Therefore, I'd like to use the class.weights parameter of the e1071 R package interface to libsvm to balance the influence of the two groups in the training data.

Since I was unsure exactly how these weights should be specified, I set up a little test:

1. Generate some null data (random features; 2:1 ratio between group labels)
2. Fit an svm with the class.weights parameter set.
3. Predict a bunch of new null datasets and look at the class proportions.
4. Replicate the whole process many times for different null training sets.

Here is the R code I'm using:

nullSVM <- function(n.var, n.obs) {
# Simulate null training data
vars   = matrix(rnorm(n.var*n.obs), nrow=n.obs)
labels = rep(c('a', 'a', 'b'), length.out=n.obs)
data   = data.frame(group=labels, vars)

# Fit SVM
fit = svm(group ~ ., data=data, class.weights=c(a=0.5, b=1))

# Calculate the average fraction of 'a' we would predict from null test data
mean(replicate(50, table(predict(fit, data.frame(matrix(rnorm(n.var*n.obs), nrow=n.obs))))[1])) / n.obs
}

library(e1071)
set.seed(12345)
mean(replicate(50, nullSVM(50, 300)))


From this whole thing I was expecting an output ~ 0.5, however, that's not what I got:

> mean(replicate(50, nullSVM(50, 300)))
[1] 0.6429987


The class.weights paramter is working, sort of, as the lower I weight a, the lower it is represented in this simulation (and if I omit class.weights it returns close to 1)...but I do not understand why simply using weights of 1:2 (for training data that is 2:1) does not get me all the way down to 50%.

If I'm misunderstanding SVMs, can someone explain this point? (or send some refs?)

If I'm doing it wrong, can someone tell me the correct way to use the class.weights parameter?

Could it possibly be a bug? (I think not, since I understand this software and the underlying libsvm to be quite mature)

-

 This makes sense. When I reduce the number of features and increase the number of observations in this simulation, the output value moves closer to 0.5. However, it never quite gets there - even with 900 rows and only 1 column. – John Colby Nov 5 '11 at 17:06 Of course on real data I always use the caret package or the built-in tune() function for model parameter tuning, so I especially like your second idea for how to deal with this in practice by adjusting the resampling scheme to favor the minority class. Much appreciated. – John Colby Nov 5 '11 at 17:08 glad you suggestion was useful. There is a paper on setting the optimal ratio which might also be useful theoval.cmp.uea.ac.uk/publications/pdf/ijcnn2001.pdf However, the optimal theoretical correction isn't always optimal in practice, so the best results might actually be obtained by tuning the two separate C parameters without forcing a particular ratio, but weighting the patterns according to class when evaluating the leave-one-out model selection criterion. – Dikran Marsupial Nov 5 '11 at 17:43 I'd also add, these days I tend to use kernel ridge regression rather than SVMs as you don't have these sort of counter-intuitive problems due to the discontinuity in the derivative of the loss function. Quite often if you tune an L2 SVM properly, you end up with a very small value of C and all the data are SVs, at which point you have a KRR model anyway. The more I used them, the less useful I have found SVMs in practice, although the theoretical insights they have brought have been vital. – Dikran Marsupial Nov 5 '11 at 17:47