I have been running some trials for recommendations using Collaborative Filtering, specifically Alternating Least Squares (ALS).
I am using two versions of ALS, one with fixed lambda regularisation and one using weighted-lambda regularisation (as seen in this paper)
The results are some what confusing to me. Using the MovieLens 1m dataset, split 60/20/20 for train/test/validate with rank 12, lambda = 0.1 with 10 iterations I get the following results:
Fixed Lambda RMSE (train) = 0.7139 RMSE (test) = 1.0206 Weighted Lambda RMSE (train) = 1.3865 RMSE (test) = 0.8792
The fixed lambda results make sense to me, the RMSE is higher on the test set as its out of sample.
However, with weighted regularisation no matter what rank factors I use to train the model, the RMSE is always higher on the train set when using weighted regularisation. Can someone perhaps explain, or point in the direction a paper, to explain why this would happen? This goes against everything I know of model building.