2
$\begingroup$

I'm working with the Anderson Iris data set and it is too small To split into a test and training set.I use boosting To make a classifier For determining the species Of flower Based on Variables in The data set.

I'd like to use cross validation To Test my predictor.My understanding is that I need to make a for loop That runs The function On all but one Of the observations.

Is there a function I can use that automate this process? Am I right That Cross validation can be used To test The error rate for my boosting tree?

this is my boosting tree

library('adabag')
boost <- boosting(Species~.,data=ii,boos = TRUE, mfinal=3)
$\endgroup$
  • $\begingroup$ While I haven't used it myself, I believe the caret package can be used to automate such tasks $\endgroup$ – dsaxton Jan 4 at 1:13
  • $\begingroup$ Cross validation depends on the nature of your data (in order to achieve independence with the splitting), but not on the chosen model. You can actually set it up with black-box functions for training and prediction. (With the slight exception that for certain training algorithms and particular cross validation procedures it is possible to obtain more efficient reformulated computations - but these are typically possible only for very easy "textbook" situations). So from that point of view the answer is: as always. $\endgroup$ – cbeleites supports Monica Jan 7 at 18:06
1
$\begingroup$

Yes, you are correct that cross-validation can be used; it actually recommended that we cross-validate our results to get a better understanding of their out-of-sample performance as well as their variability.

As mentioned in the comments by @dsaxton the package caret has a lot of convenient functionality when it comes to training a model and getting performance estimates through resampling approaches (bootstrap, repeated $k$-fold cross-validation, LOOCV, etc.). Check the functions caret::createDataPartition and caret::createFolds for more details. Specific to LOOCV, we can easily implement it ourselves; for the data shown we can simply do something like:

U <- sapply(seq(nrow(iris)), function(myindex){
       mybooster <- boosting(Species ~ ., data = iris[-indx,], boos = TRUE, mfinal = 3)
       mypreds <-predict(mybooster, newdata = iris[indx,])$class 
       mypreds == iris[indx,"Species"] # <- simplistic label comparison/only for demo
})

where we would directly get an Accuracy-like metric in the vector U. (In general, I would strongly urge you to view classification in terms of assigning a probability instead of a label but that is another question.)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.