1
$\begingroup$

I have been running some trials for recommendations using Collaborative Filtering, specifically Alternating Least Squares (ALS).

I am using two versions of ALS, one with fixed lambda regularisation and one using weighted-lambda regularisation (as seen in this paper)

The results are some what confusing to me. Using the MovieLens 1m dataset, split 60/20/20 for train/test/validate with rank 12, lambda = 0.1 with 10 iterations I get the following results:

Fixed Lambda
RMSE (train) = 0.7139
RMSE (test) = 1.0206

Weighted Lambda
RMSE (train) = 1.3865
RMSE (test) = 0.8792

The fixed lambda results make sense to me, the RMSE is higher on the test set as its out of sample.

However, with weighted regularisation no matter what rank factors I use to train the model, the RMSE is always higher on the train set when using weighted regularisation. Can someone perhaps explain, or point in the direction a paper, to explain why this would happen? This goes against everything I know of model building.

Improve this question
$\endgroup$
4
  • $\begingroup$ Maybe I am missing something but I think what you observe is expected. There is no guarantee that a smaller RMSE will be obtained. If anything you get a smaller test-error when using the weighted lambda-approach so it seems you get better generalisation (which is good). $\endgroup$
    – usεr11852
    Commented Mar 31, 2016 at 19:51
  • $\begingroup$ I understand that using the weighted lambda-approach will result in a smaller test-error, but I didn't expect it to be lower than the train-error? When I use a larger dataset (MovieLens 20m) this does not happen. $\endgroup$
    – statisticnewbie12345
    Commented Mar 31, 2016 at 19:54
  • $\begingroup$ Why wouldn't be lower? It is a random partition after-all. In general the accuracy on a training set is a bit meaningless aside detecting overfitting. For example, if you have some outliers in your training error those would drive the RMSE high but the testing error RMSE would be lower. $\endgroup$
    – usεr11852
    Commented Mar 31, 2016 at 20:10
  • $\begingroup$ You can try making a plot of the test and training error across values of $\lambda$; that often is quite informative. $\endgroup$
    – usεr11852
    Commented Mar 31, 2016 at 20:13

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.