5
$\begingroup$

Using R, I have developed three models:

  • linear regression using lm();
  • decision tree using rpart();
  • k-nearest neighbor using kknn().

I would like to conduct leave-one-out cross-validation tests and compare these models. However, which error metric should I use for better representation? Does mean absolute percentage error (MAPE) or sMAPE (symmetric MAPE) look fine? Please suggest me a metric.

For example, when I conducted leave-one-out CV tests on linear regression (LR) and decision tree (DT) models, the sMAPE error values are 0.16 and 0.20. However, the R-squared values of LR and DT are 0.85 and 0.92 respectively. Where sMAPE computed as [sum (abs(predicted - actual)/((predicted + actual)/2))] / (number of data points). Here DT is pruned regression tree. These R^2 values are computed on full data set. There are a total of 60 data points in the set.

Model  R^2   sMAPE
 LR    0.85   0.16
 DT    0.92   0.20
Improve this question
$\endgroup$
3
  • 2
    $\begingroup$ The metric will depend on the purpose of the regression. Typically, though, because most regressions work by minimizing something (like a sum of squares of residuals or a negative log likelihood), you usually would choose a metric similar to that objective function. Otherwise your regression probably is inappropriate. $\endgroup$
    – whuber ♦
    Commented Jul 25, 2011 at 21:40
  • $\begingroup$ @kpp Please, feel free to link your questions when they are related to the same problem (unless I am wrong). $\endgroup$
    – chl
    Commented Jul 25, 2011 at 21:44
  • 2
    $\begingroup$ @kkp, this article by R. Hyndman and A. Koeller has a comparison of various fit metrics. It might be of use for you. $\endgroup$
    – mpiktas
    Commented Jul 26, 2011 at 6:49

1 Answer 1

Reset to default
4
$\begingroup$

Lots of metric exist and no one is generally the best to use, it depends of your problem, of your data. Often, many metric can be used. I find usefull, to compute both hypothesis test and different metric (RMSE, MAPE ...), and see if they provide similar result. So your conclusions won't be based only on one metric.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.