R caret Naive Bayes (untuned) results differ from klaR

Question

I'm running a naive bayes classification model and I noticed that the caret package returns a different result than does klaR (which caret references) or e1071.

My question is: is there something I'm doing wrong with my caret syntax that I'm not able to recover the same results as with klaR (or e1071)?

Please note that I am running an untuned model in caret and providing it with the same specifications that I'm providing to klaR (usekernal = FALSE and fL = 0).

In the example below, both e1071 and klaR return identical confusion matrices (which makes sense given that klaR is based on e1071 but adds the kernal and laplace smoother, which I've disabled here). Oddly, when caret is asked to run an untuned model with the same specification as the klaR model, the results are close but not identical but I would expect the results to be identical to klaR.

Here's a reproducible example:

# Load Libraries
library(kernlab); #for spam data
library(caret)
library(e1071)
library(klaR)

# Load Data
data(spam)

# e1071 naiveBayes
set.seed(3456)
fit1 <- naiveBayes(spam, spam$type, type="raw")
    pred1 <- predict(fit1, spam, type="class")
    confusionMatrix(pred1, spam$type)

# klaR NaiveBayes
set.seed(3456)
fit2 <- NaiveBayes(spam, spam$type, usekernal = FALSE, fL = 0)
    pred2 <- predict(fit2, spam)
    #Warnings that probability is 0 for some cases
    confusionMatrix(pred2$class, spam$type)

# caret with no tuning, usekernal = FALSE, fL = 0
set.seed(3456)
fit3 <- train(type ~ ., 
         data=spam,
         method = "nb",
         trControl = trainControl(method="none"),
         tuneGrid = data.frame(fL=0, usekernel=FALSE))

pred3 <- predict(fit3, spam, type="raw")
#Warnings that probability is 0 for some cases
confusionMatrix(pred3, spam$type)

Here are selected outputs from the confusion matrices.

For e1071:

Accuracy : 0.7266
Sensitivity : 0.5814          
Specificity : 0.9498         

For klaR:

Accuracy : 0.7266
Sensitivity : 0.5814          
Specificity : 0.9498 

For caret:

Accuracy : 0.7135
Sensitivity : 0.5610          
Specificity : 0.9482 

Any information on why this is happening and what, if anything, I can do about it, is greatly appreciated.

Thanks!

EDIT: Just in case this is helpful, from sessionInfo()

R version 3.2.2 (2015-08-14)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 8 x64 (build 9200)

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252    LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C                           LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] klaR_0.6-12     MASS_7.3-45     e1071_1.6-7     caret_6.0-58    ggplot2_1.0.1   lattice_0.20-33 kernlab_0.9-22 

loaded via a namespace (and not attached):
 [1] Rcpp_0.12.1        magrittr_1.5       splines_3.2.2      munsell_0.4.2      colorspace_1.2-6   foreach_1.4.3     
 [7] minqa_1.2.4        stringr_1.0.0      car_2.1-0          plyr_1.8.3         tools_3.2.2        parallel_3.2.2    
[13] nnet_7.3-11        pbkrtest_0.4-2     grid_3.2.2         gtable_0.1.2       nlme_3.1-122       mgcv_1.8-7        
[19] quantreg_5.19      class_7.3-14       MatrixModels_0.4-1 iterators_1.0.8    lme4_1.1-10        digest_0.6.8      
[25] Matrix_1.2-2       nloptr_1.0.4       reshape2_1.4.1     codetools_0.2-14   stringi_1.0-1      scales_0.3.0      
[31] combinat_0.0-8     stats4_3.2.2       SparseM_1.7        proto_0.3-10  

Answer 1

The problem lies in the fact that you use a different specification in the models. In fit1 and fit2 you use the x and y combination, in fit3 the formula notation

If you switch all models in the formula notation (type ~ ., data = spam) you will see an Accuracy of 0.7135

If you switch all models in the x / y notation (spam, spam$type) you will see an Accuracy of 0.7266

There is probably someone who can explain why exactly this difference occurs. I have no idea except that it has something to do in the difference of how the S3 formula notation is processed versus the default notation of x and y.

Answer 2

From my knowledge, for the first two models, you shouldn't be giving the whole spam data frame as the training variables (the class labels are considered as a feature in this case). Instead, you should use:

fit1 <- naiveBayes(spam[,-58], spam$type, type="raw")

this way it will produce the same results with (type ~., data=spam).

Answer 3

The difference is whether x includes the full matrix including class variable, or whether y is specifically excluded from x.

# y included in x -> Accuracy : 0.7266

fit3 <- train(
   x=spam,
   y=spam$type,
   method = "nb",
   trControl = trainControl(method="none"),
   tuneGrid = data.frame(usekernel=FALSE,fL=0,adjust=FALSE))

pred3 <- predict(fit3, spam, type="raw")
#Warnings that probability is 0 for some cases
confusionMatrix(pred3, spam$type)
#Accuracy : 0.7266

is different to:

# y excluded from x -> #Accuracy : 0.7135

fit3 <- train(
   x=spam[,-dim(spam)[2]],
   y=spam$type,
   method = "nb",
   trControl = trainControl(method="none"),
   tuneGrid = data.frame(usekernel=FALSE,fL=0,adjust=FALSE))

pred3 <- predict(fit3, spam, type="raw")
#Warnings that probability is 0 for some cases
confusionMatrix(pred3, spam$type)
#Accuracy : 0.7135