I have a machine learning model that has 6 parameters (of various dimensions). When I run stochastic gradient ascent on my code, one of those 6 parameters has a (relatively) huge gradient. I believe this is why my model doesn't learn. This one parameter has a huge enough gradient to demand a small learning rate, but then consequently the other parameters don't move far enough in the appropriate direction... and this one parameter alone doesn't provide enough information for my model to learn.

My question, therefore, is -- what is the best way to deal with this issue? Am I stuck using a fancier optimizer than SGD, and if so, is there a suggested one (Adam?). Or is there a simpler solution to this problem?

EDIT: Using something like Adam isn't great, actually. This is a recommender system using implicit data: see this paper. Only a tiny subset of parameters change every time. Specifically, each training example is a triple of (user, item1, item2). Each user and item has its own parameter associated with it. So obviously, for a given training example, the gradient will be zero for all items besides item1 and item2; likewise for all users besides user.

Adam have me continually updating old items while they get 0 gradient for many thousands of iterations (there are thousands of users and items). While it'll decay quick enough, it doesn't seem like the right tool to use for my problem.

Are there any other ways of reducing the gradient in one direction and not others -- in a more principled manner than manually changing the learning rate for one term?

  • 1
    $\begingroup$ Are the features scaled? $\endgroup$
    – User0
    Sep 4, 2017 at 23:17
  • $\begingroup$ I updated the question, because the old one left out important information. This isn't a typical classifier. It's a recommender system with implicit feedback. Input is simply a (user, item1, item2) triple. However, each item is associated with visual features (from a VGG-16 neural net) which isn't scaled. I could look into normalizing the network's outputs (e.g. to [0-1]) but it's not clear why [0-1] is an appropriate scale (the median of the network outputs is already ~0.1). $\endgroup$
    – anon
    Sep 4, 2017 at 23:41
  • $\begingroup$ I would argue that scaling your learning rate so the parameters have similar gradients is a form of normalization and is quite "principled". $\endgroup$
    – combo
    Sep 5, 2017 at 1:08
  • $\begingroup$ ^ I appreciate the feedback. Having a different learning rate does make logical sense to me, so I'll give it a try tomorrow. But I'm quite curious about your particular response. How would you say it's normalization/ principled? I'm not disagreeing with you -- I'm totally clueless on the matter. $\endgroup$
    – anon
    Sep 5, 2017 at 11:15
  • 1
    $\begingroup$ Yeah, I'm familiar with the above. However, in this case I'm not normalizing my inputs... I'm explicitly changing the step size in a particular direction. As far as I know, that's not something that is often done. Anyway, I tested just using different learning rates and empirically it seems to work. After 150K iterations, I hit an AUC of 0.8. In contrast, the non-visual portion needs ~3M iterations to get that high. I need to test LR annealing strategies and do more hyperparameter tuning, So that's cool. $\endgroup$
    – anon
    Sep 5, 2017 at 21:31

1 Answer 1


I ended up doing what I mentioned in the OP: simply giving different terms different learning rates. I'm still not sure if it's mathematically justified, but empirically it worked great. Results were quite good without much care in hyperparameter tuning or learning rate annealing -- and it got there much quicker than with the non-visual component.

So there you go. If you know a priori one parameter needs a smaller learning rate than the others to train, then don't hesitate making the learning rate non-uniform.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.