I'm using different models to model count data, the purpose of modelling is prediction. Values vary from 0 to 7. I try to use cross-validation method to assess out-of-sample predictive perfomance, but what error measure should I use? Is RMSE enough? What other methods of models comparing and assessment can I use?
$\begingroup$
$\endgroup$
1
-
1$\begingroup$ I listed nice alternatives for performance metrics here stats.stackexchange.com/questions/71720/… Here are some graphical methods listed stats.stackexchange.com/questions/70558/…, instead of RSME $\endgroup$– MomoCommented Aug 4, 2014 at 9:02
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
RMSE is definitely the first thing that comes to mind. It punishes large deviations more than small ones, and its size is meaningful in terms of the underlying variable.
You can also define your own error function, if that is not suitable for your needs. Say, if small errors of 1-2 are not important when the baseline is large, you could define some sort of relative error (say $|y_i - \hat y_i|/y_i$).
Overall I would suggest you use a measure that is both familiar, and ideally the same measure you use to train your model. RMSE should fit both those criteria, unless you have a strong reason to prefer something else.
-
$\begingroup$ Thanks for your anwser. And what about absolute assesment? As I understood RMSE, MAE, correlation coefficients between observed and predicted values, cross-validation are all more or less relative and used mostly for comparison. And how can I assess a predictive perfomance of a single model? $\endgroup$ Commented Apr 30, 2014 at 8:21
-
$\begingroup$ It's impossible to answer this question in general. The error measure you use should represent the cost of that error that you are trying to minimize. If you only care about hits/misses you could use something akin to classification error: $\sum_i^n I(y_i \neq \hat y_i)/n$ where all errors, large or small, count the same. What is the underlying problem you are solving? $\endgroup$– ilirCommented Apr 30, 2014 at 8:30