0
$\begingroup$

Is it okay to concatenate a neural network training process repetitively using different sets of hyperparameters?

I am asking because I made a mistake when trying to optimize the hyperparameters of a vanilla NN. Instead of create a new network instance every iteration for each set of hyperparameters, I put the initiation function out of the loop (so only one instance is created).

So the initial weights for next iter training process (with a new set of hyperparameters) is using the weights trained based on last set of hyperparameters. However, doing this gives me higher validation accuracy.

My first thought was the results improved because of better/smarter weight initialization. But I still don't feel this is a sensible approach. Any ideas or comments about why and why not to do so are appreciated!

$\endgroup$
0
$\begingroup$

As long as you dont train on your test set at any time, any approach is valid. You'll measure the effectiveness of your approach using your dev or test set.

What you probably want to do to see if you are getting a benefit from the approach above (which is totally possible, since neural nets are highly empirical, theory lags our results considerably...), is to use cross-validation.

Take your training set, set aside eg 20% as a test set. Dont touch this. Then run eg 5-fold or 10-fold cross-validation using the remaining 80% of your test set. In each fold, you'll divide this remaining 80% into 80% train, and 20% dev, run your full procedure above, measure the accuracy. Repeat for the other folds. Take the average accuracy. Is it better/worse/same as what you originally intended to do?

There's no obvious way to predict the results of doing this without trying it, in general.

$\endgroup$
  • $\begingroup$ Hi Hugh, thanks for the answer! I didn't incorporate cross validation in this practice since my compute resource is quite limited and the network takes quite long to run. Without CV my validation accuracy did build up after iterations of different sets of hyperparameters. My observation is that fixing learning rate, with a (probably) better weight initialization and adding slightly a bit regularization each iteration improves the result. $\endgroup$ – Ritaotao Feb 6 '18 at 3:36
0
$\begingroup$

As Hugh Perkins mentioned, you approach is valid. Just adding to him, you've make sure that the results are re-producible everytime with the same instanciation.

As long as you don't train on test/val data & able to provide re-producible result any training approach is valid.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.