# Running SGD when one parameter's gradient is significantly larger than the rest

I have a machine learning model that has 6 parameters (of various dimensions). When I run stochastic gradient ascent on my code, one of those 6 parameters has a (relatively) huge gradient. I believe this is why my model doesn't learn. This one parameter has a huge enough gradient to demand a small learning rate, but then consequently the other parameters don't move far enough in the appropriate direction... and this one parameter alone doesn't provide enough information for my model to learn.

My question, therefore, is -- what is the best way to deal with this issue? Am I stuck using a fancier optimizer than SGD, and if so, is there a suggested one (Adam?). Or is there a simpler solution to this problem?

EDIT: Using something like Adam isn't great, actually. This is a recommender system using implicit data: see this paper. Only a tiny subset of parameters change every time. Specifically, each training example is a triple of (user, item1, item2). Each user and item has its own parameter associated with it. So obviously, for a given training example, the gradient will be zero for all items besides item1 and item2; likewise for all users besides user.

Adam have me continually updating old items while they get 0 gradient for many thousands of iterations (there are thousands of users and items). While it'll decay quick enough, it doesn't seem like the right tool to use for my problem.

Are there any other ways of reducing the gradient in one direction and not others -- in a more principled manner than manually changing the learning rate for one term?

• Are the features scaled? – user0 Sep 4 '17 at 23:17
• I updated the question, because the old one left out important information. This isn't a typical classifier. It's a recommender system with implicit feedback. Input is simply a (user, item1, item2) triple. However, each item is associated with visual features (from a VGG-16 neural net) which isn't scaled. I could look into normalizing the network's outputs (e.g. to [0-1]) but it's not clear why [0-1] is an appropriate scale (the median of the network outputs is already ~0.1). – anon Sep 4 '17 at 23:41
• I would argue that scaling your learning rate so the parameters have similar gradients is a form of normalization and is quite "principled". – combo Sep 5 '17 at 1:08
• ^ I appreciate the feedback. Having a different learning rate does make logical sense to me, so I'll give it a try tomorrow. But I'm quite curious about your particular response. How would you say it's normalization/ principled? I'm not disagreeing with you -- I'm totally clueless on the matter. – anon Sep 5 '17 at 11:15
• Yeah, I'm familiar with the above. However, in this case I'm not normalizing my inputs... I'm explicitly changing the step size in a particular direction. As far as I know, that's not something that is often done. Anyway, I tested just using different learning rates and empirically it seems to work. After 150K iterations, I hit an AUC of 0.8. In contrast, the non-visual portion needs ~3M iterations to get that high. I need to test LR annealing strategies and do more hyperparameter tuning, So that's cool. – anon Sep 5 '17 at 21:31