I am trying to evaluate my recommender system which uses Non-negative Matrix Factorization. Some things that I evaluate are
- How does the size of the feature matrix affect the recommendations
- How does the amount of "rated movies" affect the recommendation
- ...
Should I evaluate these things with a usual trainingset-testset separation? Like 60% Trainingset and 40% Testset? What is about the fact that every factorization of NNMF leads a bit different results, because of non-convexity and random initialization? I would use a random 60/40 separation, but I thought I´d better ask before I do it wrong.
And if I would decide to make several runs of NNMF in order to select the best results, then I would need to do cross validation right? Or can I use the same training/testset for every run?
