Why there is decrease in AUC values with increase in number of iteration in Held out validation?

Question

I have a dataset with 600 rows and 4000 columns for which I am trying to do held-out cross-validation with 10 and 100 iterations.

At first, the dataset is split into 80%:20% training and “held-out” test sets. Five-fold cross-validation was done. This algorithm was then used for prediction on the “held-out” 20% data in the test set. This process was repeated N = 10/100 times.

For held-out, with an increase in the number of iterations, the AUC curve is going down. For 10, iterations I am getting a mean AUC of 0.80, whereas for 100 iterations I am getting a mean AUC of 0.67.

What could be the possible reason for this decrease in AUC values?

Is it due to the parameter tunning?

train_control <- trainControl(method="cv", number=10)

svm.model <- svm(Class ~ ., data = training,metric="ROC", ranges=list(cost=10^(-1:2), gamma=c(.5,1,2)), type="eps",trControl=train_control,kernel="radial",na.action=na.omit,probability = TRUE)

Is it the right notation to get the list of cost and gamma values?

Answer 1

Having more columns that rows is a really bad omen. If your data have more dimensions than samples your classification model will be doomed to generalization failure, I suppose in your case increasing number of iteration will increase overfitting. I suggest you at first: try to reduce column number or greatly increase number of samples.