I´m doing data classification using SGB algorithm. First I divided my dataset into training (80%) and test (20%) subsets. I used the training dataset to train and tune the SGB parameters and evaluate its performance using 10 fold cross-validation. Based on this approach I select the best SGB model and used this to predict new values in the test subset to evaluate its accuracy calculating a confusion matrix. However, I would like to now if it is possible to obtain a confusion matrix based on cross-validation using the test subset. The reason is that some classes have few observation, therefore.
I used the caret R package to perform the SGB classification:
# data partition: set.seed(3456) tab_x<-createDataPartition(tab_a$code,p=0.80,list=FALSE,times=1) train_set<-tab_a[tab_x,] test_set<-tab_a[-tab_x,] # tuning model SGB (Stochastic Gradient Boosting) fitControl<-trainControl(method="repeatedcv",number=10,repeats=3) sgbGrid_x<-expand.grid(interaction.depth=c(1,3,5,9,11),n.trees=(1:30)*50,shrinkage=0.01) nrow(sgbGrid_x)
"code" represents 8 categories and "X1, X2, X3, X4, X5,X6,X7" are predictor variables
sgbFit<-train(code~X1+X2+X3+X4+X5+X6+X7,data=train_set,method="gbm",trControl=fitControl,bag.fraction=0.50,verbose=FALSE,tuneGrid=sgbGrid_x) #fitting the best SGB model (founded during the previous steps) sgbGrid_a<-expand.grid(interaction.depth=5,n.trees=550,shrinkage=0.05) nrow(sgbGrid_a) sgbFit_a<-train(code~X1+X2+X3+X4+X5+X6+X7,data=train_set,method="gbm",trControl=fitControl,verbose=FALSE,tuneGrid=sgbGrid_a) #calculating confusion matrix using test samples classif<-predict(sgbFit_a,newdata=test_set,type="raw") cm_sgb<-confusionMatrix(classif,test_set$code) cm_sgb
Since some categories have few observations in the test subset, I think this is better to use a k-fold cross validation to produce a confusion matrix, but how can I do this in R? Please can anyone help me with the R code?