I'm using the 100k MovieLense dataset to build a recommendation system in R, using the recommenderlabpackage.

From what I understood, the recommenderlab package kind of forces the user to use the following evaluation technique.

  • Split the dataset into training and test sets.
  • Split the test set into known and unknown, base in the given argument

This is what's done when I run the following:

MovieLense <- binarize(MovieLense, minRating=4)
MovieLense <- MovieLense[rowSums(MovieLense) > 4,]

eva <- evaluationScheme(
  data = MovieLense,
  method = "split",
  train = 0.75, 
  given = 3 

So I my folowing work is:

  • Train the recommender using the traing set
  • Predict rating for users in the test set using the known data
  • Evaluate prediction accuracy in the unknown test set data

This is performed using:

rec <- Recommender(getData(eva, "train"), method = "IBCF")
pred <- predict(rec, getData(eva, "known"), type = "topN", n = 5)
error <- calcPredictionAccuracy(pred, getData(eva, "unknown"), given = 3)

After all this, my question is:

  • Is this a good evaluation technique?

I'm not confortable in using always only 3 given items to predict users ratings, because I think the algorithm could have a better performance if I used more ratings in prediction.

But, I know that if I use more "given" items, my precision will be smaller because there will be less relevant items too.

So, how do I choose the best number of items?

If this is a wrong way the evaluate my system:

  • What's the best way to measure my algorithm performance?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.