3
$\begingroup$

I am working on neural networks for a regression problem in R using packages like nnet, caret etc. I have split my data into train, validation and test. My doubt is does the train() function in caret package for R takes care for validation set also.

From what I understand, After training the nnet model, you need to keep checking with validation data set, to avoid overfitting or overlearning i.e restricting the number of iterations. Then we have to tune for the decay parameters and size of hidden layers and finally apply it on the test data set.

Is there anything wrong with the understanding? FYI. Here is the code that I am implementing

Y=read.csv(file="./dolcan.csv",header=T)
ratio=as.integer(0.5*nrow(Y))
ratio1=as.integer(0.75*nrow(Y))
traindata=Y[(1:ratio),c(2:ncol(Y))]
valdata=Y[(ratio:ratio1),c(2:ncol(Y))]
testdata=Y[((ratio1+1):nrow(Y)),c(2:ncol(Y))]

## Train the network and tuning the number of nodes and decay
maxout= max(traindata[,1]) # to scale the output
mygrid <- expand.grid(.decay=c(0.5, 0.1), .size=c(3,4,5))
nnetfit <- train(dolcan/maxout ~ ., data=traindata, method="nnet", maxit=1000, tuneGrid=mygrid, trace=F)
nnetfit
$\endgroup$
  • $\begingroup$ Additionally whenever I run the code, The RMSE produced is different inspite of the same data set being used. $\endgroup$ – NG_21 Dec 6 '13 at 10:41
2
$\begingroup$

Your understanding, while completely legitimate, is fairly "old school". For example, early stopping was used to prevent overfitting before weight decay, which turned out to be a more elegant and effective technique. Also, monitoring the error rate with a single, separate validation set has been supplanted (IMO) by model tuning via resampling (e.g. cross-validation or the bootstrap). That said, there are a lot of different philosophies and opinions about how things should be done.

I don't think that you need a validation set and a test set. I would use resampling (as you are) to see how the model is doing on the training set and then, if you think that you've finalized the model, verify the results on the test set.

If you are modeling a continuous outcome, you might think about using a linear function between the hidden layer and the outcome (e.g. the nnet option linout).

Also, you will get repeatable results if you fix the random number seed before the call to train (see ?set.seed).

Max

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ My only concern is how well would I know if I am overfitting, for example the error keeps on decreasing in the training set as I increase the size of nodes in the layer which is obvious. $\endgroup$ – NG_21 Dec 10 '13 at 5:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.