# Why RMSE is too small?

Sorry, I'm a newbie at recommender systems, but I wrote few lines of code using the Apache Mahout library.

My dataset is pretty small, 500x100 with 8102 cells known.

The dataset is actually a subset of the Yelp dataset from the "Yelp business rating prediction" competition. I just took the top 100 most commented restaurants, and then took the 500 most active customers.

I created SVDRecommender and then I evaluated RMSE.

The result is about 0.4... Why is it so small?

Maybe I just don't understand something and my dataset is not so sparse, but then I tried with a larger and more sparse dataset and RMSE became even smaller (about 0.18)! Could anyone explain this behaviour?

DataModel model = new FileDataModel(new File("datamf.csv"));
final RatingSGDFactorizer factorizer = new RatingSGDFactorizer(model, 20, 200);
final Factorization f = factorizer.factorize();

RecommenderBuilder builder = new RecommenderBuilder() {
public Recommender buildRecommender(DataModel model) throws TasteException {
return new SVDRecommender(model, factorizer);
}
};

RecommenderEvaluator evaluator = new RMSRecommenderEvaluator();
double score = evaluator.evaluate(builder,
null,
model,
0.6,
1);

System.out.println(score);

• RMSE is relative measure, i.e. it is expressed in the scale of your variable (see e.g. here). So there is no "objectively small" value for RMSE, it depends on your data. So you have to tell us why do you consider this value to be "too small"? What is your data? – Tim Jun 20 '15 at 9:04
• @Tim. Well, yeah, thanks, i thought that RMSE is an absolute measure. But how can I interpret this result? Should I calculate the correlation coefficient? My data, as I said, is a subset of Yelp dataset 500x100 with 8102 cells defined. Each cell is a rating from 1 to 5. – luckyfish Jun 20 '15 at 9:18
• You use measures such as RMSE to compare and/or adjust models for the same data, so if you have only one model and one RMSE value does not really tell you much. – Tim Jun 20 '15 at 9:26
• @Tim As I understand, RMSE strongly depends on data? I thought that RMSE is an absolute value, so i can say that "my recommender predict ratings with rmse_value error". and now, as i understand, I should talk "my recommender predicts ratings with this error on this data"? What are the data characteristics RMSE depends on (size, sparsity or scale of rating values)? (sorry for my bad english) – luckyfish Jun 20 '15 at 9:50
• Check the link I posted in in the first comment - in describes how RMSE is computed and so tells you on what it depends exactly. I would also recommend you come book on machine learning, see stats.stackexchange.com/questions/143402/… or stats.stackexchange.com/questions/18973/… for references. – Tim Jun 20 '15 at 10:04