0
$\begingroup$

I'm using the caret package's train function to optimize a neural net using nnet. I'm new to neuralnet modelling so this is a learning process for me.

My data has 17 input fields, about 10,000 records, and I'm trying to predict a binary outcome.

Using a grid search, I'm scanning a range of nodes from 3:16 and a range of around five decay values between 0.1 and 0.001.

The issue I have is that the training process always results in the maximum number of nodes possible being selected. This gives a good cross-validation Kappa score and high AUC, but it generalizes poorly to a holdout set, which feels like a classic case of overfitting.

My question is this: I believe the decay parameter should be set to penalize high node counts, but in a grid scan I'm simply determining the best fit - which means a low decay and high node count. Should I be approaching this differently: e.g. by fixing a decay value to determine a node count, then flexing the decay value in a second phase? What point does a parameter grid scan have here if it can minimise the "penalty" ?

$\endgroup$
-1
$\begingroup$

I would actully use some kind of cross-validation or leave-one-out approach to determine the error rate and the most appropriate node-decay combination.

It is a difficult question though. I usually set the decay first after I scanned the literature for reasonable values to this specific problem. Then I vary the nodes by looking at the test errors e.g. with LOOCV...

/Tobias

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.