2
$\begingroup$

I'm using different models to model count data, the purpose of modelling is prediction. Values vary from 0 to 7. I try to use cross-validation method to assess out-of-sample predictive perfomance, but what error measure should I use? Is RMSE enough? What other methods of models comparing and assessment can I use?

$\endgroup$
1
$\begingroup$

RMSE is definitely the first thing that comes to mind. It punishes large deviations more than small ones, and its size is meaningful in terms of the underlying variable.

You can also define your own error function, if that is not suitable for your needs. Say, if small errors of 1-2 are not important when the baseline is large, you could define some sort of relative error (say $|y_i - \hat y_i|/y_i$).

Overall I would suggest you use a measure that is both familiar, and ideally the same measure you use to train your model. RMSE should fit both those criteria, unless you have a strong reason to prefer something else.

$\endgroup$
  • $\begingroup$ Thanks for your anwser. And what about absolute assesment? As I understood RMSE, MAE, correlation coefficients between observed and predicted values, cross-validation are all more or less relative and used mostly for comparison. And how can I assess a predictive perfomance of a single model? $\endgroup$ – Evgenii Nikitin Apr 30 '14 at 8:21
  • $\begingroup$ It's impossible to answer this question in general. The error measure you use should represent the cost of that error that you are trying to minimize. If you only care about hits/misses you could use something akin to classification error: $\sum_i^n I(y_i \neq \hat y_i)/n$ where all errors, large or small, count the same. What is the underlying problem you are solving? $\endgroup$ – ilir Apr 30 '14 at 8:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.