How does one deal with the issue where a group of parameters moves at a good pace, while others are stalled during training in Deep Learning?

Question

I was noticing that some of my parameters where training faster than others. However, I was not sure of the source of this nor how to deal with it. The only thing that occurred to me is to have a different learning rate per parameters but that seems very hacky and difficult to scale (i.e. a human has to intervene for every change in the network and data set etc).

Has the Deep Learning community developed a better tool, algorithm etc to deal with such a problem?

Answer 1

What optimization algorithm are you using for training? Variants of SGD such as RMSProp and ADAM, (among many other variants) are designed to update all parameters at approximately the same rate. They do so exactly as you describe -- by giving each parameter a different learning rate. I suggest using those algorithms if you are not already.