in a derivation for the gradient of a collaborative filtering system (similar to Probabilistic Matrix Factorization), I got to the following expression for the gradient of a latent vector $\mathbf{u}_i$:

$\frac{\partial}{\partial \mathbf{u}_i} = 2\sum_{j=1}^M \left[ (\mathbf{u}_i^T\mathbf{v}_j - R_{ij})\mathbf{v}_j \right] + \lambda_u (\mathbf{u}_i - \boldsymbol \mu_u)$

The gradient is a sum of terms, hence I can apply SGD! But I wonder, how should I treat the term that is outside the summation? Should I estimate $M$ and then add it to the individual $j$-gradients multiplied by $1/M$? Should I use it only every $M$ updates? Should I use it randomly or with a precise schema?

What would your choice be, and why?


SGD is shown to converge (under certain strong assumptions) to local minima of the loss $L$ as long as the expected value (under the distribution from which you pick samples) of the update is equal to $\nabla L$.

So yes, your idea of splitting the additional term should work. Just rewrite your full gradient as a sum by dividing the new term by $1/M$ ($M$ being the total number of samples in the training set) and under the same assumptions as without it, you will have that the expected value of the update is the full gradient.

For references see Leon Bottou's papers, in particular his introduction to SGD.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.