How to get generalisation performance from nnet in R using k-fold cross-validation?

Question

I'm doing some Machine Learning in R using the nnet package. I want to estimate the generalisation performance of my classifier by using k-fold cross-validation.

How should I go about doing this? Are there some built-in functions that do this for me? I've seen tune.nnet() in the e1071 package, but I'm not sure that does quite what I want.

Basically I want to do the cross-validation (split into 10 groups, train on 9, test on the other 1, repeat) and then from that obtain some sort of measure of how well my classifier generalises - but I'm not sure what measure that should be. I guess I want to look at the average of the accuracies across the different cross-validation examples, but I'm not sure how to do that with the tune.nnet() function above.

Any ideas?

Answer 1

If you are planning to tune the network (e.g. select a value for the learning rate) on your training data and determine the error generalization on that same data set, you need to use a nested cross validation, where in each fold, you are tuning the model on that 9/10 of the data set (using 10 fold cv). See this post where I asked a similar question and got very good answers (post).

I am not sure if there is an R function to accomplish this - maybe the ipred package could be used by passing the tune function with the call - not sure. It is rather trivial to simply write a loop to accomplish this whole process though.

As far as what metric to examine, that depends on your problem (classification versus regression) and if you are interested in accuracy (classifications rate or kappa) or how well the model ranks (e.g. lift) or the MAE or RMSE for regression.

Answer 2

Implementing k-fold CV (with or without nesting) is relatively straightforward in R; and stratified sampling (wrt. class membership or subjects' characteristics, e.g. age or gender) is not that difficult.

About the way to assess one's classifier performance, you can directly look at the R code for the tune() function. (Just type tune at the R prompt.) For a classification problem, this is the class agreement (between predicted and observed class membership) that is computed.

However, if you are looking for a complete R framework where data preprocessing (feature elimination, scaling, etc.), training/test resampling, and comparative measures of classifiers accuracy are provided in few commands, I would definitely recommend to have a look at the caret package, which also includes a lot of useful vignettes (see also the JSS paper).

Of note, although NNs are part of the methods callable from within caret, you may probably have to look at other methods that perform as well and most of the times better than NNs (e.g., Random Forests, SVMs, etc.)