Neural network for prediction

Question

I am working on neural networks for a regression problem in R using packages like nnet, caret etc. I have split my data into train, validation and test. My doubt is does the train() function in caret package for R takes care for validation set also.

From what I understand, After training the nnet model, you need to keep checking with validation data set, to avoid overfitting or overlearning i.e restricting the number of iterations. Then we have to tune for the decay parameters and size of hidden layers and finally apply it on the test data set.

Is there anything wrong with the understanding? FYI. Here is the code that I am implementing

Y=read.csv(file="./dolcan.csv",header=T)
ratio=as.integer(0.5*nrow(Y))
ratio1=as.integer(0.75*nrow(Y))
traindata=Y[(1:ratio),c(2:ncol(Y))]
valdata=Y[(ratio:ratio1),c(2:ncol(Y))]
testdata=Y[((ratio1+1):nrow(Y)),c(2:ncol(Y))]

## Train the network and tuning the number of nodes and decay
maxout= max(traindata[,1]) # to scale the output
mygrid <- expand.grid(.decay=c(0.5, 0.1), .size=c(3,4,5))
nnetfit <- train(dolcan/maxout ~ ., data=traindata, method="nnet", maxit=1000, tuneGrid=mygrid, trace=F)
nnetfit

Answer 1

Your understanding, while completely legitimate, is fairly "old school". For example, early stopping was used to prevent overfitting before weight decay, which turned out to be a more elegant and effective technique. Also, monitoring the error rate with a single, separate validation set has been supplanted (IMO) by model tuning via resampling (e.g. cross-validation or the bootstrap). That said, there are a lot of different philosophies and opinions about how things should be done.

I don't think that you need a validation set and a test set. I would use resampling (as you are) to see how the model is doing on the training set and then, if you think that you've finalized the model, verify the results on the test set.

If you are modeling a continuous outcome, you might think about using a linear function between the hidden layer and the outcome (e.g. the nnet option linout).

Also, you will get repeatable results if you fix the random number seed before the call to train (see ?set.seed).

Max