2
$\begingroup$

I want to compare three models, one linear-regression-model, one regression-tree-model (from rpart) and one MARS-model (from mda package).

I want to compare the models using a leave one out cross validation using the mean square error and MAPE. I have the following implementation in R:

library(data.table)
library(rpart)
library(mda)

#Load Sample-Data
data(trees)

#The following models should be compared:
# lm(Volume~Girth+Height, data=trees)
# rpart(Volume~Girth+Height, data=trees)
# mars(trees[,-3], trees[3])

LOOCV<-function(modelCall) {
  unlist(sapply(seq(1,nrow(trees)), function(i) {         
    training=trees[-i,]
    test=trees[i,]

    fit=eval(modelCall)
    testValue = predict(fit, test[1:2])

    test[3]-testValue
  }))
}

LOOCV_MSE<-function(modelCall) {
   sum(LOOCV(modelCall)^2)/nrow(trees)
}

LOOCV_RMSE<-function(modelCall) {
   sqrt(LOOCV_MSE(modelCall))
}

LOOCV_MAPE<-function(modelCall) {
  sum(abs(LOOCV(modelCall)/sapply(seq(1, nrow(trees)), function(i) {trees[i,3]})))/nrow(trees)*100                                    
}


cat("Cross-Validation Metrics:\n")
cat("-------------------------\n")
cat("LOOCV MSE for LM:", LOOCV_MSE(quote(lm(Volume~Girth+Height, data=training))),"\n")
cat("LOOCV MSE for CART:", LOOCV_MSE(quote(rpart(Volume~Girth+Height, data=training))),"\n")
cat("LOOCV MSE for MARS:", LOOCV_MSE(quote(mars(training[,-3], training[3]))),"\n")
cat("\n")

cat("LOOCV RMSE for LM:", LOOCV_RMSE(quote(lm(Volume~Girth+Height, data=training))),"\n")
cat("LOOCV RMSE for CART:", LOOCV_RMSE(quote(rpart(Volume~Girth+Height, data=training))),"\n")
cat("LOOCV RMSE for MARS:", LOOCV_RMSE(quote(mars(training[,-3], training[3]))),"\n")
cat("\n")

cat("LOOCV MAPE for LM:", LOOCV_MAPE(quote(lm(Volume~Girth+Height, data=training))),"\n")
cat("LOOCV MAPE for CART:", LOOCV_MAPE(quote(rpart(Volume~Girth+Height, data=training))),"\n")
cat("LOOCV MAPE for MARS:", LOOCV_MAPE(quote(mars(training[,-3], training[3]))),"\n")

Outputs:

Cross-Validation Metrics:
-------------------------
LOOCV MSE for LM: 18.15783 
LOOCV MSE for CART: 69.83769 
LOOCV MSE for MARS: 13.72282 

LOOCV RMSE for LM: 4.2612 
LOOCV RMSE for CART: 8.356895 
LOOCV RMSE for MARS: 3.704432 

LOOCV MAPE for LM: 14.6114 
LOOCV MAPE for CART: 23.51401 
LOOCV MAPE for MARS: 10.00316 

Does this implementation make sense? When whould using MSE on the errors make sense? When would I use MAPE/SMAPE instead? I already read "Metric to compare models?" and the conclusion there was it depends, can someone explain this further. On what does it depend?

My data is not a time series, it is more like the tree example data.

$\endgroup$
6
$\begingroup$

Why re-invent the wheel? R already has many libraries that implement cross-validation and calculate RMSE, MSE, MAPE, etc:

library(caret)
library(forecast)
library(rpart)
library(mda)

#Load Sample-Data
data(trees)

#Custom Summary Function for Cross-Validation
customSummary <- function (data, lev = NULL, model = NULL) {
    stats1 <- postResample(data[, "pred"], data[, "obs"])
    stats2 <- accuracy(data[, "pred"], data[, "obs"])
    c(stats1, stats2)
}

#Choose sampling method and sumamry function
myControl <- trainControl(method='LOOCV', summary=customSummary)

#Run Models
model_LM <- train(Volume~Girth+Height, data=trees, method='lm', 
                  trControl=myControl)
model_CART <- train(Volume~Girth+Height, data=trees, method='rpart', 
                  trControl=myControl)
model_MARS <- train(Volume~Girth+Height, data=trees, method='earth', 
                  trControl=myControl)

#Compare models
model_LM
model_CART
model_MARS

Choosing which statistics to use is up to you, and depends on the particulars of your problem. I usually use MAE as a default, but it pays to think about what the actual error cost in your problem.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.