Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 0 down vote favorite

ok, so I have a memory-based model to predict the values of variable X and I've also measured the real values of X. I have calculated both the RMSE and MAPE for the difference Xreal-Xpredicted. I know that there is absolutely no point in saying a model is good or bad if the RMSE value is less than a particular value. I was wondering though if there is a rule of thumb providing insights on the subject. For example my data range from 0 to 4 and I have an RMSE of 1,1. Is there any empirical rule suggesting this is a good/bad estimation?

thank you in advance

ETA: X may take all values within the range{-2,2}

share|improve this question
1  
The ratio of the RMSE to the standard deviation of the data is directly related to $R^2$, so for (extensive) discussion about this point, please review questions on r-squared such as stats.stackexchange.com/questions/13314/… and stats.stackexchange.com/questions/14585. – whuber♦ Apr 23 '12 at 18:38
thank you. However the prediction is not the result of a regression process therefore I was wondering if some other rules apply to it. – Anastasia Apr 23 '12 at 18:59
1  
I am curious why we should care how a model is derived when we are assessing its quality. (Putting aside the problems with assessing model fit using the same data employed in computing that fit...). It seems to me that if you view RMSE as a valid measure of fit, then that's all that matters: the inner workings of the model should be immaterial. – whuber♦ Apr 23 '12 at 19:32
What is the nature of your outcome, $X$? Is it a continuous, ordinal, polytomous, or even binary variable? My concern is whether classification accuracy could be an issue. It's always useful to, at least, describe what metric you're trying to predict. In general, there is no global measure of predictive accuracy. The most adequate substitute is a prespecified minimally sufficient, expected, and superior level of predictive accuracy based on input from an expert in the field. This is a different story if you have a competing predictive model. – AdamO Apr 23 '12 at 20:01
@AdamO: yes, X is a continuous variable. I just wonder whether, since it takes values within a certain range, that would give us any insight regarding the quality of estimation. I mean, if we have and RMSE of 2 for the range [-2,2] is quite different from having the same RMSE for [-100,100], no? – Anastasia Oct 22 '12 at 13:59

Know someone who can answer? Share a link to this question via email, Google+, Twitter, or Facebook.

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.