You can use the sperrorest
package to estimate the performance of your model. Various error measures are returned.
The following code is taken from ?sperrorest
. You are welcome to modify pred.fun
to your needs. This is just an example.
The actual code syntax refers to version 1.0.0 which is currently available from Github using devtools::install_github("pat-s/sperrorest")
.
data(ecuador) # Muenchow et al. (2012), see ?ecuador
fo <- slides ~ dem + slope + hcurv + vcurv + log.carea + cslope
# Example of a classification tree fitted to this data:
library(rpart)
ctrl <- rpart.control(cp = 0.005) # show the effects of overfitting
fit <- rpart(fo, data = ecuador, control = ctrl)
par(xpd = TRUE)
plot(fit, compress = TRUE, main = 'Stoyans landslide data set')
text(fit, use.n = TRUE)
# Non-spatial 5-repeated 10-fold cross-validation:
mypred.rpart <- function(object, newdata) predict(object, newdata)[, 2]
nspres <- sperrorest(data = ecuador, formula = fo,
model.fun = rpart, model.args = list(control = ctrl),
pred.fun = mypred.rpart,
smp.fun = partition.cv,
smp.args = list(repetition = 1:5, nfold = 10))
summary(nspres$error.rep)
summary(nspres$error.fold)
mean sd median IQR
train.auroc 7.465106e-01 0.005186850 7.478738e-01 0.003922117
train.error 2.953111e-01 0.005235886 2.952222e-01 0.004166667
train.accuracy 7.046889e-01 0.005235886 7.047778e-01 0.004166667
train.sensitivity 6.742444e-01 0.011529458 6.727778e-01 0.016000000
train.specificity 7.351333e-01 0.005994030 7.361111e-01 0.007111111
train.fpr70 3.115111e-01 0.017166685 3.202222e-01 0.023777778
train.fpr80 4.801778e-01 0.022528528 4.900000e-01 0.035222222
train.fpr90 6.819556e-01 0.015305853 6.808889e-01 0.019555556
train.tpr80 5.599111e-01 0.016436447 5.530000e-01 0.023000000
train.tpr90 3.345556e-01 0.013286259 3.297778e-01 0.022444444
train.tpr95 1.702667e-01 0.017200632 1.728889e-01 0.004666667
train.events 9.000000e+03 0.000000000 9.000000e+03 0.000000000
train.count 1.800000e+04 0.000000000 1.800000e+04 0.000000000
test.auroc 6.446438e-01 0.008173371 6.446755e-01 0.003872000
test.error 3.863000e-01 0.004590752 3.850000e-01 0.006500000
test.accuracy 6.137000e-01 0.004590752 6.150000e-01 0.006500000
test.sensitivity 5.890000e-01 0.018179659 5.960000e-01 0.021000000
test.specificity 6.384000e-01 0.012601587 6.350000e-01 0.020000000
test.fpr70 5.156000e-01 0.020634922 5.190000e-01 0.034000000
test.fpr80 6.286000e-01 0.018447222 6.360000e-01 0.017000000
test.fpr90 7.910000e-01 0.018547237 7.880000e-01 0.008000000
test.tpr80 3.978000e-01 0.009364828 3.970000e-01 0.004000000
test.tpr90 1.844000e-01 0.012239281 1.870000e-01 0.014000000
test.tpr95 8.560000e-02 0.020767763 8.800000e-02 0.025000000
test.events 1.000000e+03 0.000000000 1.000000e+03 0.000000000
test.count 2.000000e+03 0.000000000 2.000000e+03 0.000000000
ph1
? Please add a reproducible example for people to work with. $\endgroup$