I have a dataset with 600 rows and 4000 columns for which I am trying to do held-out cross-validation with 10 and 100 iterations.

At first, the dataset is split into 80%:20% training and “held-out” test sets. Five-fold cross-validation was done. This algorithm was then used for prediction on the “held-out” 20% data in the test set. This process was repeated N = 10/100 times.

For held-out, with an increase in the number of iterations, the AUC curve is going down. For 10, iterations I am getting a mean AUC of 0.80, whereas for 100 iterations I am getting a mean AUC of 0.67.

What could be the possible reason for this decrease in AUC values?

Is it due to the parameter tunning?

train_control <- trainControl(method="cv", number=10)

svm.model <- svm(Class ~ ., data = training,metric="ROC", ranges=list(cost=10^(-1:2), gamma=c(.5,1,2)), type="eps",trControl=train_control,kernel="radial",na.action=na.omit,probability = TRUE)

Is it the right notation to get the list of cost and gamma values?

  • $\begingroup$ Can you show a plot of your AUC values through iterations? For both the model and the test set. $\endgroup$ – user2974951 Feb 14 at 13:01

Having more columns that rows is a really bad omen. If your data have more dimensions than samples your classification model will be doomed to generalization failure, I suppose in your case increasing number of iteration will increase overfitting. I suggest you at first: try to reduce column number or greatly increase number of samples.

  • $\begingroup$ Thanks @podludek, that seems to be tough in case of the dataset I am handling. $\endgroup$ – sp2 Feb 14 at 10:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.