Gaudette and Japkowicz 2009 compared various metrics for ordinal classification accuracy and they showed that, as a single statistic, the RMSE (root mean squared error) or MSE (mean squared error) performed better than the other measures that they found in the literature. Although RMSE/MSE is designed for continuous data, its property of penalizing deviations from the mean more severely works well for ordinal data converted to small integers.

However, Baccianella et al 2009 showed that MAE (mean absolute error) performed very poorly for measuring performance when the ordinal categories was imbalanced in real-life data that they tested; they also implied that MSE also performs poorly. (They mentioned, though, that in an artificial dataset, the performance difference was not as severe.) So, they proposed an adapted measure which they called macroaveraged MAE that gives equal weight to all categories, thus nullifying the effects of imbalance. As far as I understand it, their adaptation basically calculates the MAE one category at a time and then takes the average of all categories, giving each category equal weight: see the article for details. However, they also showed that their adapted version of MAE was mathematically identical to the regular version when the categories were balanced.

So, based on these two articles, I recommend that you attempt to use Baccianella's adapted version of MAE or MSE in general, especially if your target variable categories are significantly imbalanced. However, if the categories are balanced, then the simple RMSE or MSE should be a good measure, and might be preferred for its simplicity.

Gaudette and Japkowicz 2009 compared various metrics for ordinal classification accuracy and they showed that, as a single statistic, the RMSE (root mean squared error) or MSE (mean squared error) performed better than the other measures that they found in the literature. Although RMSE/MSE is designed for continuous data, its property of penalizing deviations from the mean more severely works well for ordinal data converted to small integers.

However, Baccianella et al 2009 showed that MAE (mean absolute error) performed very poorly for measuring performance when the ordinal categories were imbalanced in real-life data that they tested; they also implied that MSE also performs poorly. (They mentioned, though, that in an artificial dataset, the performance difference was not as severe.) So, they proposed an adapted measure which they called macroaveraged MAE that gives equal weight to all categories, thus nullifying the effects of imbalance. As far as I understand it, their adaptation basically calculates the MAE one category at a time and then takes the average of all categories, giving each category equal weight: see the article for details. However, they also showed that their adapted version of MAE was mathematically identical to the regular version when the categories were balanced.

So, based on these two articles, I recommend that you attempt to use Baccianella's adapted version of MAE or MSE in general, especially if your target variable categories are significantly imbalanced. However, if the categories are balanced, then the simple RMSE or MSE should be a good measure, and might be preferred for its simplicity.