1
$\begingroup$

I have a dataset with many subjects, and for each subject I have 100 samples for an X variable, while Y has exactly 50%/50% cases for label 1/2. I am trying to run cross-validation with SVM and found that accuracies are always below 50% despite the set being balanced.

So I made up the following simulation and changed the sample size (see two figures for 50 and 1000). Still, I get the same below 50% bias. Changing k has an effect, and increasing the sample size also helps - but this doesn't go away - SVM keeps giving me a mean accuracy below 50%, over samples. Cost doesn't make a difference as well as far as I can tell.

Any thoughts???

Thanks, Nitzan

library(e1071)

nsamples=100
N=2000
x<-runif(nsamples,0,1)
y<-array(1:2, dim=c(nsamples))
z<-array(0,dim=c(N))

for (i in 1:N){  
x<-x[sample(length(x))] 
y<-y[sample(length(y))]
model=svm(x,y,
          type='C-classification',
          kernel="linear",cross=20)
z[i]=mean(model$accuracies)}

histogram(z,nint=20,xlim=c(0,100))
mean(z)

enter image description here

$\endgroup$
1
$\begingroup$

Figured this one out. The problem was that from some reason svm did not use stratification for cross validation...

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.