0
$\begingroup$

I am having trouble understanding the output of caret::train( ) with method = "glmStepAIC". Here is the sample code on the dataset PimaIndiansDiabetes. There are 8 predictor variables insulin, age, pressure, pedigree, pregnant, mass, glucose, triceps. The response variable is binary diabetes.

library(mlbench)
data(PimaIndiansDiabetes)

library(caret)
trControl <- trainControl(method = "repeatedcv",
                          repeats = 3,
                          classProbs = TRUE,
                          number = 10, 
                          savePredictions = TRUE,
                          summaryFunction = twoClassSummary)

  caret_model <- train(diabetes~., 
                       data=PimaIndiansDiabetes, 
                       method="glmStepAIC", 
                       family = "binomial",
                       direction ="backward",
                       trControl=trControl)

The output I got is very long, an extract is here:

Start:  AIC=679.72
.outcome ~ pregnant + glucose + pressure + triceps + insulin + 
    mass + pedigree + age

           Df Deviance    AIC
- triceps   1   661.75 677.75
- insulin   1   662.17 678.17
<none>          661.72 679.72
- age       1   664.75 680.75
- pressure  1   669.92 685.92
- pedigree  1   670.43 686.43
- pregnant  1   673.20 689.20
- mass      1   695.17 711.17
- glucose   1   759.66 775.66

Step:  AIC=677.75
.outcome ~ pregnant + glucose + pressure + insulin + mass + pedigree + 
    age

           Df Deviance    AIC
- insulin   1   662.19 676.19
<none>          661.75 677.75
- age       1   664.75 678.75
- pressure  1   670.04 684.04
- pedigree  1   670.62 684.62
- pregnant  1   673.22 687.22
- mass      1   699.39 713.39
- glucose   1   761.45 775.45

Step:  AIC=676.19
.outcome ~ pregnant + glucose + pressure + mass + pedigree + 
    age

           Df Deviance    AIC
<none>          662.19 676.19
- age       1   665.53 677.53
- pressure  1   670.62 682.62
- pedigree  1   670.71 682.71
- pregnant  1   673.74 685.74
- mass      1   699.39 711.39
- glucose   1   768.91 780.91
Start:  AIC=644.26
.outcome ~ pregnant + glucose + pressure + triceps + insulin + 
    mass + pedigree + age

           Df Deviance    AIC
- triceps   1   626.39 642.39
- insulin   1   627.76 643.76
<none>          626.26 644.26
- age       1   628.72 644.72
- pressure  1   631.71 647.71
- pedigree  1   635.02 651.02
- pregnant  1   641.49 657.49
- mass      1   665.00 681.00
- glucose   1   740.27 756.27

Step:  AIC=642.39
.outcome ~ pregnant + glucose + pressure + insulin + mass + pedigree + 
    age

           Df Deviance    AIC
<none>          626.39 642.39
- insulin   1   628.67 642.67
- age       1   628.95 642.95
- pressure  1   632.33 646.33
- pedigree  1   635.04 649.04
- pregnant  1   641.55 655.55
- mass      1   668.08 682.08
- glucose   1   743.92 757.92
Start:  AIC=741.45
.outcome ~ pregnant + glucose + pressure + triceps + insulin + 
    mass + pedigree + age

           Df Deviance    AIC
- triceps   1   723.45 739.45
- insulin   1   725.19 741.19
<none>          723.45 741.45
- age       1   725.97 741.97
- pressure  1   729.99 745.99
- pedigree  1   733.78 749.78
- pregnant  1   738.68 754.68
- mass      1   764.22 780.22
- glucose   1   838.37 854.37

Step:  AIC=739.45
.outcome ~ pregnant + glucose + pressure + insulin + mass + pedigree + 
    age

           Df Deviance    AIC
<none>          723.45 739.45
- insulin   1   725.46 739.46
- age       1   725.97 739.97
- pressure  1   730.13 744.13
- pedigree  1   733.92 747.92
- pregnant  1   738.69 752.69
- mass      1   768.77 782.77
- glucose   1   840.87 854.87

I understand that if a predictor increases the AIC of the model, it should not be included in the model. From the output, how do I know what are the final list of variables that are kept? I can see that triceps is no longer at the bottom of the output, is it already discarded?

Furthermore, if I inspect variable importance using varImp(caret_model). I see this :

ROC curve variable importance

         Importance
glucose     100.000
mass         59.818
age          59.567
pregnant     32.626
pedigree     27.306
pressure     19.418
triceps       6.299
insulin       0.000

Why is insulin the least important but it is still included in the model after training?

$\endgroup$
  • $\begingroup$ Also a side thought: Is it because I do 3-repeated 10-fold cross-validation that the output is super long? $\endgroup$ – hydradon Jan 4 at 17:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.