What are good metrics to use to measure the error of ordinal classification?
For example, assume that we have 3 classes "LOW", "MEDIUM", and "HIGH". We should find that if we misclassify "LOW" as "MEDIUM" the error is less than if we misclassify "LOW" as "HIGH".
Are there any metrics that account for this?
Gaudette and Japkowicz 2009 compared various metrics for ordinal classification accuracy and they showed that, as a single statistic, the RMSE (root mean squared error) or MSE (mean squared error) performed better than the other measures that they found in the literature. Although RMSE/MSE is designed for continuous data, its property of penalizing deviations from the mean more severely works well for ordinal data converted to small integers.
However, Baccianella et al 2009 showed that MAE (mean absolute error) performed very poorly for measuring performance when the ordinal categories were imbalanced in real-life data that they tested; they also implied that MSE also performs poorly. (They mentioned, though, that in an artificial dataset, the performance difference was not as severe.) So, they proposed an adapted measure which they called macroaveraged MAE that gives equal weight to all categories, thus nullifying the effects of imbalance. As far as I understand it, their adaptation basically calculates the MAE one category at a time and then takes the average of all categories, giving each category equal weight: see the article for details. However, they also showed that their adapted version of MAE was mathematically identical to the regular version when the categories were balanced.
So, based on these two articles, I recommend that you attempt to use Baccianella's adapted version of MAE or MSE in general, especially if your target variable categories are significantly imbalanced. However, if the categories are balanced, then the simple RMSE or MSE should be a good measure, and might be preferred for its simplicity.
Jianlin Cheng, A Neural Network Approach to Ordinal Regression, 2007 and Niu et al., Ordinal Regression with Multiple Output CNN for Age Estimation, 2016 utilize a clever representation of the labels to measure error with cross entropy.
They present the the total error as the sum of errors in predicting whether or not the "rank" of a sample $x_i$ is greater than rank $k_i$.
In other words, we would generate predictions of vectors with elements $r(x_i) > k_i$, representing the prediction of the classifier for whether or not the rank of the sample is greater than each rank. This becomes a multiclass classification problem and error functions for that problem can be utilized. Total error, then, can be considered a sum of the individual binary classifier loss functions (such as cross-entropy).
E.g., Predicted rank = 2 results in a predicted vector = [1, 0, 0]. Actual rank = 3 results in a label vector = [1, 1, 0]. Then calculate loss between each prediction in the vector.
Another explanation of this method can be found here.
This problem can be converted to regular regression by simply casting the ranks of the respective labels into integers. E.g., "HIGH" = 2, "MEDIUM"=1, "LOW"=0. Regression loss functions can then be used.
