4
$\begingroup$

I am experimenting with neural network package in R nnet and i have some questions.

  1. The regulatory environment i am working on requires me to reproduce my results to show them to the auditors. How can i reproduce my model results after few months/years ? Can i use a seed value to control the model output ?
  2. How can i validate a neural network model ? are there any goodness of fit tests ?
  3. How do i choose the number of hidden layers i need ? I have 18500 observations in my training dataset and 8 variables. Does that help in identifying hidden layers required in anyway?
  4. Many times the model stops after 100 iterations. I have used maxiter option but sometimes you see the output and it says converged and sometimes it says stopped after 100 iterations. When it says stopped after 100 iterations does that mean i have a bad model ? and it did not converge ?

Thank you

$\endgroup$

1 Answer 1

7
$\begingroup$
  1. Setting a seed is a good start. It should ensure that you will always get the same results from the same code. It would also be worth noting the version of R and the version of nnet that you're using (as well as any other relevant packages), in case something changes in later versions. If you want to be really careful, you could even archive the current versions so you can go back to them if CRAN ever goes down.

  2. The best evaluation approach usually involves some kind of out-of-sample prediction. Cross-validation is one very good option, as is a simple holdout approach. The caret package automates some of these tools. You have so much data that holding some of it out probably won't hurt you much.

  3. nnet always has one hidden layer. Perhaps you meant the number of hidden units? Cross-validation can be a good option here as well. Just make sure that you don't use the same data for improving the model fit that you plan on using to evaluate model performance (as in question #2).

  4. Convergence means that the parameters aren't changing anymore. It means that the nnet has done as well as it can, given the setup you chose and the random seed used. If you want it to run for more iterations, try increasing the size of the hidden layer or starting from a different set of initial conditions.

$\endgroup$
4
  • $\begingroup$ 1. I referred to this cran.r-project.org/web/packages/nnet/nnet.pdf and there is no mention of setting seed as an option. Can you tell me how to do that ? 2. Thank you i will try that. 3. Great take away. I did not know that. are there any examples of cross validation in R that i can read ? 4. I used maxiter option to run it for more iterations but i get stopped after 100 iterations ( my question is does that mean the model did not converge and we have a bad model ? ) Thanks $\endgroup$
    – user16789
    May 21, 2013 at 21:00
  • 1
    $\begingroup$ 1) use the set.seed function before starting the model. 3) the caret and cvTools packages both automate some aspects of cross-validation and both have vignettes/manuals with examples. 4) I'm not quite sure what's going on there. $\endgroup$ May 21, 2013 at 23:01
  • $\begingroup$ David, setting seed=arandomnumber are we going to compromise on global minimum in any way ? $\endgroup$
    – user16789
    May 22, 2013 at 0:30
  • 1
    $\begingroup$ nnet is trying to solve a very non-convex optimization problem, so if your net has more than a few hidden nodes, you'll probably never find the global minimum, regardless of what seed you use. $\endgroup$ May 22, 2013 at 19:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.