I'm using the
caret package's train function to optimize a neural net using
nnet. I'm new to neuralnet modelling so this is a learning process for me.
My data has 17 input fields, about 10,000 records, and I'm trying to predict a binary outcome.
Using a grid search, I'm scanning a range of nodes from 3:16 and a range of around five decay values between 0.1 and 0.001.
The issue I have is that the training process always results in the maximum number of nodes possible being selected. This gives a good cross-validation Kappa score and high AUC, but it generalizes poorly to a holdout set, which feels like a classic case of overfitting.
My question is this: I believe the decay parameter should be set to penalize high node counts, but in a grid scan I'm simply determining the best fit - which means a low decay and high node count. Should I be approaching this differently: e.g. by fixing a decay value to determine a node count, then flexing the decay value in a second phase? What point does a parameter grid scan have here if it can minimise the "penalty" ?