1
$\begingroup$

Sorry, I'm a newbie at recommender systems, but I wrote few lines of code using the Apache Mahout library.

My dataset is pretty small, 500x100 with 8102 cells known.

The dataset is actually a subset of the Yelp dataset from the "Yelp business rating prediction" competition. I just took the top 100 most commented restaurants, and then took the 500 most active customers.

I created SVDRecommender and then I evaluated RMSE.

The result is about 0.4... Why is it so small?

Maybe I just don't understand something and my dataset is not so sparse, but then I tried with a larger and more sparse dataset and RMSE became even smaller (about 0.18)! Could anyone explain this behaviour?

DataModel model = new FileDataModel(new File("datamf.csv"));
final RatingSGDFactorizer factorizer = new RatingSGDFactorizer(model, 20, 200);
final Factorization f = factorizer.factorize();


RecommenderBuilder builder = new RecommenderBuilder() {
        public Recommender buildRecommender(DataModel model) throws TasteException {
            return new SVDRecommender(model, factorizer);
        }
    };


RecommenderEvaluator evaluator = new RMSRecommenderEvaluator();
    double score = evaluator.evaluate(builder,
            null,
            model,
            0.6,
            1);

System.out.println(score);
Improve this question
$\endgroup$
5
  • $\begingroup$ RMSE is relative measure, i.e. it is expressed in the scale of your variable (see e.g. here). So there is no "objectively small" value for RMSE, it depends on your data. So you have to tell us why do you consider this value to be "too small"? What is your data? $\endgroup$
    – Tim
    Commented Jun 20, 2015 at 9:04
  • $\begingroup$ @Tim. Well, yeah, thanks, i thought that RMSE is an absolute measure. But how can I interpret this result? Should I calculate the correlation coefficient? My data, as I said, is a subset of Yelp dataset 500x100 with 8102 cells defined. Each cell is a rating from 1 to 5. $\endgroup$
    – luckyfish
    Commented Jun 20, 2015 at 9:18
  • $\begingroup$ You use measures such as RMSE to compare and/or adjust models for the same data, so if you have only one model and one RMSE value does not really tell you much. $\endgroup$
    – Tim
    Commented Jun 20, 2015 at 9:26
  • $\begingroup$ @Tim As I understand, RMSE strongly depends on data? I thought that RMSE is an absolute value, so i can say that "my recommender predict ratings with rmse_value error". and now, as i understand, I should talk "my recommender predicts ratings with this error on this data"? What are the data characteristics RMSE depends on (size, sparsity or scale of rating values)? (sorry for my bad english) $\endgroup$
    – luckyfish
    Commented Jun 20, 2015 at 9:50
  • $\begingroup$ Check the link I posted in in the first comment - in describes how RMSE is computed and so tells you on what it depends exactly. I would also recommend you come book on machine learning, see stats.stackexchange.com/questions/143402/… or stats.stackexchange.com/questions/18973/… for references. $\endgroup$
    – Tim
    Commented Jun 20, 2015 at 10:04

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.