0
$\begingroup$

I'm using the R h2o package for building a deep net with three hidden layers. When inspecting the model object, I'm noticing the training RMSE fluctuates as a function of number of epochs. I'm assuming with a stable gradient, the train RMSE should monotonically decrease as a function of epochs until convergence.

Are there parameters I should vary to stabilize learning as a function of epochs?

$\endgroup$
  • $\begingroup$ it is a neural network. It is a big, messy, sloppy neural network. Any of the "normal" neural network issues - it will have those and more. I found in my nearly trivial experiments with deep networks that multiplying the number of nodes on a layer by 3x can reduce oscillations at the cost of slower learning. $\endgroup$ – EngrStudent - Reinstate Monica Sep 23 '15 at 21:08
0
$\begingroup$

H2O deep learning is finding local minimum or saddle points, that's likely why you are seeing the fluctuations.

Try varying l1 and l2 regularization, input layer and per layer dropout. Those are usually a good starting point for building models that eventually converge. You will likely want to do some sort of search (grid or random search) across these hyperparameters.

The H2O booklet covers much of these items: http://h2o-release.s3.amazonaws.com/h2o/rel-slater/3/docs-website/h2o-docs/booklets/DeepLearning_Vignette.pdf

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.