Is it possible to use advance optimization(L-BFGS, Conjugate gradient) for a collaborative filtering system vs just using gradient descent? I ask this because of the need to calculate both X and theta simutaneously.
Recent work on using conjugate gradient and graph side information for matrix factorisation with clear example and code.
Here you can see how to represent the MF problem, with graph side information, in a correct form (Sylvester equation) and solve with CG https://papers.nips.cc/paper/5938-collaborative-filtering-with-graph-information-consistency-and-scalable-methods
code avaliable here https://github.com/rofuyu/exp-grmf-nips15
Absolutely, see my example in - http://sanealytics.com/2015/03/10/matrix-factorization/
You can substitute for your favorite optimization method.