1
$\begingroup$

I am applying a neural network and logistic regression to a classification problem. In order to evaluate the performance of the two classifiers I'm using 5-fold cross-validation (roughly 800 samples in the test data for each fold). Using caret I am finding the best parameters for my ANN in an inner loop in the cross-validation (roughly 600 samples in test data for each fold). However, I would really like to get a better understanding of how caret handles this.

Seeing as neural networks can benefit from retraining in order to avoid local minima I would like to do some retraining. I am not sure whether caret does this after having selected the optimal parameters or simply trains the model once on the larger sample of test data (800 samples) with the found parameters. Can anyone clarify this for me?

EDIT: And if it doesn't do any retraining, how can I do this in the best possible way?

Code:

              library(caret)
              library(nnet)
              tunGrid <-  expand.grid(size  = c(1, 2, 3, 4, 5, 6, 7),
                      decay = c(0, 10^(-2), 10^(-3), 10^(-4)))

              fitControl <- trainControl(method = "repeatedcv", number = 5, repeats=3)
              set.seed(825)
              fmla <- as.formula("y_BBR_train ~ .")  
              annFit1 <- train(fmla, data = X_BBR_train,
              method = "nnet",
              trControl = fitControl,
              tuneGrid  = tunGrid)

Thank you!

$\endgroup$

1 Answer 1

2
$\begingroup$

Let's say you have 800 samples. For each fold, you are leaving out about 160 samples and doing this 30 times. For each combination of weight decay and number of hidden units, train will use the average of the 30 resamples to figure out which is empirically best. Once that is finished, it takes those optimal settings and fits a final model with all 800 samples and that is used for future prediction.

I'm not sure if that qualifies as 'retraining' or not. You can also use model averaging with neural networks by using method = "avNNet" if you think that helps.

Max

$\endgroup$
1
  • $\begingroup$ Thank you for the explanation. When taking about 'retraining' I mean training the final model a few times. As ANN often find local minima it would likely be beneficial to retrain it. It seems from your answer that the final model is only trained once. Consequently, I will try to train it more times using the optimal settings or perhaps use the "avNNet" :) $\endgroup$
    – pir
    Jun 6, 2014 at 5:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.