# Metric to compare models?

Using R, I have developed three models:

• linear regression using lm();
• decision tree using rpart();
• k-nearest neighbor using kknn().

I would like to conduct leave-one-out cross-validation tests and compare these models. However, which error metric should I use for better representation? Does mean absolute percentage error (MAPE) or sMAPE (symmetric MAPE) look fine? Please suggest me a metric.

For example, when I conducted leave-one-out CV tests on linear regression (LR) and decision tree (DT) models, the sMAPE error values are 0.16 and 0.20. However, the R-squared values of LR and DT are 0.85 and 0.92 respectively. Where sMAPE computed as [sum (abs(predicted - actual)/((predicted + actual)/2))] / (number of data points). Here DT is pruned regression tree. These R^2 values are computed on full data set. There are a total of 60 data points in the set.

Model  R^2   sMAPE
LR    0.85   0.16
DT    0.92   0.20

• The metric will depend on the purpose of the regression. Typically, though, because most regressions work by minimizing something (like a sum of squares of residuals or a negative log likelihood), you usually would choose a metric similar to that objective function. Otherwise your regression probably is inappropriate. – whuber Jul 25 '11 at 21:40
• @kpp Please, feel free to link your questions when they are related to the same problem (unless I am wrong). – chl Jul 25 '11 at 21:44
• @kkp, this article by R. Hyndman and A. Koeller has a comparison of various fit metrics. It might be of use for you. – mpiktas Jul 26 '11 at 6:49