Converting between different accuracy/error metrics

Question

I am trying to compare model accuracy between several different measurement metrics. For example, some citations use accuracy while other use error. That one is rather obvious, but there are lots of different metrics and I am not entirely sure how to compare some of them and not lose some of the individual metrics integrity. Or whether or not some can be compared at all. The list I have is:

Error Rate - Mean Absolute Error - Absolute Error - Log-Loss - Classification Accuracy - Root Mean Squared Error - Classification Error - F-Measure - Area Under Curve - Mean Test Error - Error Percentage - Misclassification Error - Test Error - Mean Test Error

So my question is how to effectively convert between these, and if no direct conversion is possible, to compare and rank in a meaningful and accurate way.

Answer 1

We sure have plenty of terms in this fields. This wikipedia page might be a useful introduction.

The following term are used to measure accuracy in classification They measure the probability that the classifier prediction is not the same as the concept.

• Error Rate
• Classification Accuracy
• Error Percentage
• Misclassification Error

In the following measures we measure the accuracy, just as above. We add additional information saying where we measure it. Usually measures are measured on the train set or the test set. Here the accuracy is measured on the test set. - Mean Test Error - Test Error

The measures bellow are used for regression problems. In such problems the concept is usually continues and we are interested by how much the predictor missed. For example in whether classification we will be interested in knowing if it is hot or not. In regression we will be interested in knowing how hot. - Absolute Error - $abs(\hat{y}-y)$ for a specific sample y - Mean Absolute Error, the mean of the above - Root Mean Squared Error the root of the Mean Squared Error

The rest are of very different nature and here are some explanations.

• Log-Loss
• F-Measure
• The curve in "Area Under Curve" is the ROC curve

When measuring your classifiers you have different needs. These needs are answered by different measures. While if you have perfect prediction all usual measure gets their highest score, each one of them treats errors in a different way. That is way you cannot convert on into the other. For example, you might have two classifiers with the same accuracy but different F measure. For more explanation see here.