0
$\begingroup$

I have the following question. In a machine learning project I have to solve a regression and a classification task. See also: Hold-Out VS Cross-Validation - R caret

For this I have about ~650 cases available.

Since a split in training and test dataset is out of question, because the perfomance of the model would be decided by the randomness of the split, I used all data to train and did a bootstraping with 500 repetitions. This is recommended here: https://hbiostat.org/bbr/md/reg.html#internal-vs--external-model-validation

So far so good. I got RMSE, ROC AUC, sensitivity, specificity and other important parameters for my models.

But how do I report them? And is it correct to simply predicate the model against the training data at the end?

fitCtrlBootRandCf <- trainControl(method="boot",number = 500, 
                                  classProbs = TRUE, summaryFunction = twoClassSummary , search="random",
                                  savePred=T)

set.seed(1)
mGLMCf <- train(Y_Class~., data=dsCf, method="glm", family = "binomial", 
                trControl=fitCtrlCVRandCf,  
                metric="ROC", preProcess = c("center","scale"))
mGLMCf

rs <- data.frame(obs = dsCf$Y_Class,
                 pred = predict.train(mGLMCf, type="raw"),
                 prob = predict.train(mGLMCf, type="prob"))
roc.GLMCf <-  roc(rs$obs,rs$prob.classYES)
roc.GLMCf
plot(roc.GLMCf)

cm <- confusionMatrix(rs$obs,rs$pred)
round(cm$table/sum(cm$table)*100,1)
confusionMatrix(mGLMCf)

fitCtrlBootRandRg <- trainControl(method="boot", number=500, 
                                  search="random", savePredictions = T)

mGLMRg <- train(Y_Value~., data=dsRg, method="glm", 
                trControl=fitCtrlBootRandRg,  
                metric="RMSE", preProcess = c("center","scale"))
mGLMRg
RMSE(dsRg$Y_Value,predict(mGLMRg,dsRg))
plot(dsRg$Y_Value,predict(mGLMRg,dsRg))

Output:

> fitCtrlBootRandCf <- trainControl(method="boot",number = 500, 
+                                   classProbs = TRUE, summaryFunction = twoClassSummary , search="random",
+                                   savePred=T)
> 
> set.seed(1)
> mGLMCf <- train(Y_Class~., data=dsCf, method="glm", family = "binomial", 
+                 trControl=fitCtrlCVRandCf,  
+                 metric="ROC", preProcess = c("center","scale"))
> mGLMCf
Generalized Linear Model 

657 samples
 11 predictor
  2 classes: 'classYES', 'classNO' 

Pre-processing: centered (11), scaled (11) 
Resampling: Cross-Validated (10 fold, repeated 50 times) 
Summary of sample sizes: 592, 592, 591, 591, 591, 591, ... 
Resampling results:

  ROC        Sens       Spec   
  0.8291361  0.6606581  0.82225

> 
> rs <- data.frame(obs = dsCf$Y_Class,
+                  pred = predict.train(mGLMCf, type="raw"),
+                  prob = predict.train(mGLMCf, type="prob"))
> roc.GLMCf <-  roc(rs$obs,rs$prob.classYES)
Setting levels: control = classYES, case = classNO
Setting direction: controls > cases
> roc.GLMCf

Call:
roc.default(response = rs$obs, predictor = rs$prob.classYES)

Data: rs$prob.classYES in 266 controls (rs$obs classYES) > 391 cases (rs$obs classNO).
Area under the curve: 0.8419
> plot(roc.GLMCf)
> 
> cm <- confusionMatrix(rs$obs,rs$pred)
> round(cm$table/sum(cm$table)*100,1)
          Reference
Prediction classYES classNO
   classYES    27.4   13.1
   classNO     10.4   49.2
> confusionMatrix(mGLMCf)
Cross-Validated (10 fold, repeated 50 times) Confusion Matrix 

(entries are percentual average cell counts across resamples)
 
          Reference
Prediction classYES classNO
   classYES    26.7   10.6
   classNO     13.7   48.9
                            
 Accuracy (average) : 0.7568

> 
> fitCtrlBootRandRg <- trainControl(method="boot", number=500, 
+                                   search="random", savePredictions = T)
> 
> mGLMRg <- train(Y_Value~., data=dsRg, method="glm", 
+                 trControl=fitCtrlBootRandRg,  
+                 metric="RMSE", preProcess = c("center","scale"))
> mGLMRg
Generalized Linear Model 

657 samples
 11 predictor

Pre-processing: centered (11), scaled (11) 
Resampling: Bootstrapped (500 reps) 
Summary of sample sizes: 657, 657, 657, 657, 657, 657, ... 
Resampling results:

  RMSE     Rsquared   MAE     
  51.5589  0.6570902  39.82216

> RMSE(dsRg$Y_Value,predict(mGLMRg,dsRg))
[1] 50.10974
> plot(dsRg$Y_Value,predict(mGLMRg,dsRg))

enter image description here

enter image description here

$\endgroup$
0
$\begingroup$

I have now found my own answer. One can use savePrediction=T to query the predictions. Then by means of one these grouped per observation case and can plot this. If a tuning grid was used, of course only the parameters of the model$bestTune should be used.

The resulting curve corresponds in the ROC and also in the RMSE to the mean of the resamples. This is valuable because it allows one to estimate how approximately the model performs on unseen data. See also the article by Frank Hassel: https://hbiostat.org/bbr/md/reg.html#internal-vs--external-model-validation

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.