I have a machine learning model that has 6 parameters (of various dimensions). When I run stochastic gradient ascent on my code, one of those 6 parameters has a (relatively) huge gradient. I believe this is why my model doesn't learn. This one parameter has a huge enough gradient to demand a small learning rate, but then consequently the other parameters don't move far enough in the appropriate direction... and this one parameter alone doesn't provide enough information for my model to learn.
My question, therefore, is -- what is the best way to deal with this issue? Am I stuck using a fancier optimizer than SGD, and if so, is there a suggested one (Adam?). Or is there a simpler solution to this problem?
EDIT: Using something like Adam isn't great, actually. This is a recommender system using implicit data: see this paper. Only a tiny subset of parameters change every time. Specifically, each training example is a triple of (user, item1, item2). Each user and item has its own parameter associated with it. So obviously, for a given training example, the gradient will be zero for all items besides item1 and item2; likewise for all users besides user.
Adam have me continually updating old items while they get 0 gradient for many thousands of iterations (there are thousands of users and items). While it'll decay quick enough, it doesn't seem like the right tool to use for my problem.
Are there any other ways of reducing the gradient in one direction and not others -- in a more principled manner than manually changing the learning rate for one term?
