1
$\begingroup$

Looking for some help please. I have used the randomForest package extensively and have been happy with its performance. I am currently writing a journal paper investigating match outcomes in sport, predicting the outcome of matches from 33 performance indicators.

I need to use caret to produce the randomForest package as I wish to use Lime to explain outcomes in a testing data set. Lime can't accept a model from randomForest, only from a randomForest wrapped via caret.

When I use random forest I get a prediction accuracy of approx 76% on the training data set, when I tune in caret I get an accuracy of 100%.

Is this sort of improvement common? Or am I making a mistake? Would be really grateful is somebody could check my code.

My data set is named model.data

# Create model with default parameters

control <- trainControl(method="repeatedcv", number=10, repeats=3)

seed <- 7

metric <- "Accuracy"

set.seed(seed)

mtry <- sqrt(ncol(model.data))

tunegrid <- expand.grid(.mtry=mtry)

rf_group <- train(Outcome~., data=model.data, method="rf", metric=metric, tuneGrid=tunegrid, trControl=control)

print(rf_group)


# predict the outcome on the training set
Test.Predict <- predict(rf_group, model.data)
# compare predicted outcome and true outcome
confusionMatrix(Test.Predict, as.factor(model.data$Outcome))
$\endgroup$
  • 2
    $\begingroup$ 0. Welcome to the CV community. 1. Performance metrics based on the training set are misleading; avoid them. Use/report proper resampling-based measurements (e.g. like the ones caret report when using repeatedcv); report the variability of the estimates too. 2. If you are writing an article for a journal, assuming that the editor/reviewers follow simple good practices, they won't care for any metrics coming from the training set. 3. Good luck! :) $\endgroup$ – usεr11852 Mar 1 at 23:09
  • $\begingroup$ Thank you for this appreciate your time and comments $\endgroup$ – Mark Bennett Mar 1 at 23:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.