i am an R beginner and i have to do a 5 or 10-fold cross validation in a random forest model. My problem is i have to do the cv manually and not with an package. What i want to do is: 1. Building k-folds with my training data 2. Choose my tuning parameter for example trees = c( 200, 400, 600) 3. Fit my model on k-1 folds and predict my values on the holdout set(validation set) 4. Then i want to evaluate my prediction on the holdout set and save the value.

my evaluation parameter should be AUC. I understand the theory but i have problems to do this in R. Have you an idea for my code? Thanks so much!!!!

  • 1
    $\begingroup$ You seem to follow the general idea, so you need to share your code with us to guide you. $\endgroup$ – gunes Mar 28 '19 at 13:19
  • $\begingroup$ Yes exactly i don't want to do something special but i have really no idea how to do... so i have already no code, because i don't know how to start. So this is my random forest model rf = randomForest( x = training.mt.1 , y = as.factor(training.cla.mt.2) , ntree = 500 , mtry = 3 , importance = TRUE) and then i think i will beginn with something like this for(k in 1:10){.... $\endgroup$ – Theo_Mel Mar 28 '19 at 13:32

OK, here are some guidelines (a pseudocode not specific to R) to do it. A modular approach, using functions to divide up the task is best for you I think:

# 'folds' is a list having two elements [0] train-set, [1] validation-set
folds   = get_folds(data, K, <other parameters if necessary>) 
params  = <this is your hyper-parameter array>
results = <an array/or dictionary of same length as params>
ind     = 1

for param in params:
    mean_AUC = 0
    for fold in folds:
        # a train function that trains a random forest model and returns it
        # so, wrap your RF model with this method
        model = train(fold[0], param, <other parameters>)

        # an evaluate func. that gets test data and model and returns your metric 
        AUC   = evaluate(model, fold[1])
        mean_AUC += AUC

    # when all folds are finished, save your mean result
    results[ind] = mean_AUC / length(params)
    ind += 1

# when you're here, choose the best parameter wrt results array

This pseudocode should give you a start, although several optimizations can be applied (e.g. preventing data replication across folds etc.)

  • $\begingroup$ thanks for your quick response! This helps me and gives me a structure. But my problem is that i don't know how to write for explicit my train function that trains my random forest in ... do you have an example with some data? sorry for all this unprofessional questions, but i feel really lost at the moment in r. $\endgroup$ – Theo_Mel Mar 28 '19 at 13:49
  • $\begingroup$ have a look at: guru99.com/r-random-forest-tutorial.html#2 $\endgroup$ – gunes Mar 28 '19 at 13:56
  • $\begingroup$ thanks but i've already seen this, but they do it with the train function so i don't understand how to do it manual. $\endgroup$ – Theo_Mel Mar 28 '19 at 13:59
  • $\begingroup$ I can find several places in google w/o train method used: tutorialspoint.com/r/r_random_forest.htm, trevorstephens.com/kaggle-titanic-tutorial/…, datascienceplus.com/random-forests-in-r $\endgroup$ – gunes Mar 28 '19 at 14:06
  • $\begingroup$ okay thank you very much! I will have a look and try, maybe i understand it and build my model $\endgroup$ – Theo_Mel Mar 28 '19 at 14:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.