I am using neural networks for a regression problem. There is one output variable. 80 features per sample. 2000 samples total. I have 2 hidden fully connected layers with 500, and 100 neurons, respectively, both with ReLU activation. No dropout layer.

I use Adam optimization with a learning rate of learning_rate = 0.000001. I haven't tried a smaller learning rate yet, but I have tried larger ones and the oscillation was just as bad. I currently use no momentum, and no regularization. I also use the entire batch (2000 samples) for training every epoch.

As you can see in the below image, the oscillation is between 3-5 orders of magnitude. Is oscillation like this "normal" in the training process?

I have read on a few tips on resolving oscillations such as (1) Increasing the batch size (I'm already using the full batch) (2) Using a smaller learning rate (My learning rate is already pretty small, but I can try further decreasing it and see if that helps).

What are some other considerations for reducing oscillation? Should adding momentum and regularization help? Regularization, seems to be mostly for preventing overfitting, so I can't see it helping with removing oscillation. I also don't see adding momentum to be beneficial, and I think it might actually make the oscillations worse?

enter image description here

  • $\begingroup$ You probably already solved this, but try adding batch normalization before each relu (or maybe layer normalization) with a small $L_2$ weight decay and larger batch size. LR can probably be increased a bit then. $\endgroup$ – user3658307 Jun 6 at 3:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.