2
$\begingroup$

I'm wondering if I should check if a Matrix Factorization model I built for recommendation by Collaborative Filtering is overfitting.

I trained a model using MLlib ALS (Alternating Least Squares) method and it works fine for the testing data set, but on training data it was predicting almost perfectly, which is logical since I put those number into the matrix.

Is there another way to check for overfitting? Or do I not need to worry about overfitting on Matrix Factorization models?

$\endgroup$
5
  • $\begingroup$ Can you / did you cross validate? $\endgroup$ Sep 9, 2015 at 17:39
  • $\begingroup$ I can not really cross validate in a good way. I am using 90 days of history for training and then predicting on next day after training. So I could pick up any date of training to test and omit it in training data but not sure if this is what I should be doing because I would have future data in model as well. $\endgroup$ Sep 9, 2015 at 18:43
  • $\begingroup$ Why not pick a day, train the model with the 90 days before it, and predict on that day? $\endgroup$
    – Paul
    Sep 9, 2015 at 19:35
  • $\begingroup$ that is exactly what I did, but also I was training with various iteration numbers and various features number and basically as I was increasing number of features (rank param in mllib ALS) the model was getting better and better, but I would like to check model for overfitting and not sure how to do it because probably should be done differently as on regression for example. $\endgroup$ Sep 9, 2015 at 20:30
  • $\begingroup$ In my case I ignore the time completely and just remove some (user, item, rating) pairs from the training data that will be used for validation. The only thing that is important here is that the user and item are both in the training data so they can actually be used during validation. This way I was able to actually see when the model overfitted. Sorry I don't have the concrete results at hand though. What is the reason you want to test only on future data? $\endgroup$ Sep 10, 2015 at 10:16

1 Answer 1

2
$\begingroup$

The most basic thing to do if you suspect overfitting is to plot the leaning curves for the training and the test set. So train your model on a small sample, then bigger, bigger, until its the full training set. Plot error for all the test set and the subset of examples you used for training.

These curves will show if your model starts overfitting and when, also you will get a clue on the bias-variance state of your model.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.