2
$\begingroup$

I am running e1071 linear SVM on my neuroimaging data. (by function svm())

When I was doing permutation tests, I found, in average, the cross validation (CV) accuracies with shuffle labels were always below 50% (chance level) in binary classifications.

Then I created very simple datasets with random number and labels. I found the same thing. No matter what kernel (linear/RBF) I used, what cost (C) value I tried, and what CV method (10 fold, leave one out, leave two out) was applied, it is always like this. Sample size is one factor that modulates the situation a bit.

I thought when training SVM with totally random data, the CVs should fluctuate at the chance level (50%).

Here is my code for this demonstration. Did I do something wrong? I can’t figure out why.

library('ggplot2')
library('dplyr')
library('e1071')

# ------------------------- Create training function -------------------------
f_train <- function(n, nv = 2){

  # nv: number of features
  # n: number of observations


  # generate random data
  rNum = runif(nv*n)
  rNum = matrix(rNum, n, nv)
  d = as.data.frame(rNum)

  # generate random labels
  n2 = n/2
  labels = c(array(1, n2), array(0, n2))
  labels = sample(labels)
  d$condition = factor(labels)

  # training
  m_trained = svm(condition ~ ., data = d,
                  cross = 10,
                  # cross = nrow(d),  # leave one out
                  kernel = 'linear', 
                  cost = 1)

  # get CV
  acc = m_trained$tot.accuracy
  acc
}


# ------------------------------- run traiings -------------------------------
p = expand.grid(iTest = 1:100, n = seq(60, 200, 20))
data_test = p %>% group_by(iTest, n) %>% do(data.frame(acc = f_train(.$n)))

# ----------------------------------- Plot ----------------------------------- #
ggplot(data_test, aes(x = n, y = acc)) + 
  stat_summary(fun.y = "mean", colour = "red", geom = "line") + 
  stat_summary(fun.data = 'mean_sdl', geom = 'ribbon', alpha = 0.2)

enter image description here

$\endgroup$
3
$\begingroup$

your question intrigued me as I use the svm() function from this package from time to time. At first, I thought that you did run only one experiment per sample size, and were somehow unlucky with the splitting as suggested by the answer. So I re-run the experiment with your code and found the same pattern (and realized that you indeed did 100 repetitions). Then I increased the sample size to see whether the bias would disappear. Here is what I obtained with sample size of $n=10'000$ and 100 repetitions:

enter image description here

Everything seems fins, variability decreases with sample size, you seem to be around the right level of 50. But looking closely, we still have a (relatively small) down bias on average (here $\text{mean}=49.68$) which seems persistent. Looking more closely to your code, I realized that when you generate labels, you force it to be exactly balanced, it is not a random variable anymore. By changing the code to provide balanced class on expectation, now I get this enter image description here

with average of 50.00 (rounded). So to answer your question, yes there is something wrong, your variable of interest is not Bernoulli random variable. You can replace the code to generate labels with the line labels = sample.int(2,n,replace=T) (and don't forget to set the seed for reproducibility).

$\endgroup$
1
$\begingroup$

There may be some pessimistic bias, yes: If the splitting (accidentally) leads to one class being underrepresented in the training set (which will happen if the sampling is not stratified for the classes), and the classifier may be a little bit worse at recognizing it. But this class will be overrepresented in the test set. This should become less with increasing sample size.
This is described in literature for leave-one-out with small sample sizes.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.