6
$\begingroup$

I'm running a naive bayes classification model and I noticed that the caret package returns a different result than does klaR (which caret references) or e1071.

My question is: is there something I'm doing wrong with my caret syntax that I'm not able to recover the same results as with klaR (or e1071)?

Please note that I am running an untuned model in caret and providing it with the same specifications that I'm providing to klaR (usekernal = FALSE and fL = 0).

In the example below, both e1071 and klaR return identical confusion matrices (which makes sense given that klaR is based on e1071 but adds the kernal and laplace smoother, which I've disabled here). Oddly, when caret is asked to run an untuned model with the same specification as the klaR model, the results are close but not identical but I would expect the results to be identical to klaR.

Here's a reproducible example:

# Load Libraries
library(kernlab); #for spam data
library(caret)
library(e1071)
library(klaR)

# Load Data
data(spam)

# e1071 naiveBayes
set.seed(3456)
fit1 <- naiveBayes(spam, spam$type, type="raw")
    pred1 <- predict(fit1, spam, type="class")
    confusionMatrix(pred1, spam$type)

# klaR NaiveBayes
set.seed(3456)
fit2 <- NaiveBayes(spam, spam$type, usekernal = FALSE, fL = 0)
    pred2 <- predict(fit2, spam)
    #Warnings that probability is 0 for some cases
    confusionMatrix(pred2$class, spam$type)

# caret with no tuning, usekernal = FALSE, fL = 0
set.seed(3456)
fit3 <- train(type ~ ., 
         data=spam,
         method = "nb",
         trControl = trainControl(method="none"),
         tuneGrid = data.frame(fL=0, usekernel=FALSE))

pred3 <- predict(fit3, spam, type="raw")
#Warnings that probability is 0 for some cases
confusionMatrix(pred3, spam$type)

Here are selected outputs from the confusion matrices.

For e1071:

Accuracy : 0.7266
Sensitivity : 0.5814          
Specificity : 0.9498         

For klaR:

Accuracy : 0.7266
Sensitivity : 0.5814          
Specificity : 0.9498 

For caret:

Accuracy : 0.7135
Sensitivity : 0.5610          
Specificity : 0.9482 

Any information on why this is happening and what, if anything, I can do about it, is greatly appreciated.

Thanks!

EDIT: Just in case this is helpful, from sessionInfo()

R version 3.2.2 (2015-08-14)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 8 x64 (build 9200)

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252    LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C                           LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] klaR_0.6-12     MASS_7.3-45     e1071_1.6-7     caret_6.0-58    ggplot2_1.0.1   lattice_0.20-33 kernlab_0.9-22 

loaded via a namespace (and not attached):
 [1] Rcpp_0.12.1        magrittr_1.5       splines_3.2.2      munsell_0.4.2      colorspace_1.2-6   foreach_1.4.3     
 [7] minqa_1.2.4        stringr_1.0.0      car_2.1-0          plyr_1.8.3         tools_3.2.2        parallel_3.2.2    
[13] nnet_7.3-11        pbkrtest_0.4-2     grid_3.2.2         gtable_0.1.2       nlme_3.1-122       mgcv_1.8-7        
[19] quantreg_5.19      class_7.3-14       MatrixModels_0.4-1 iterators_1.0.8    lme4_1.1-10        digest_0.6.8      
[25] Matrix_1.2-2       nloptr_1.0.4       reshape2_1.4.1     codetools_0.2-14   stringi_1.0-1      scales_0.3.0      
[31] combinat_0.0-8     stats4_3.2.2       SparseM_1.7        proto_0.3-10  
$\endgroup$
8
$\begingroup$

The problem lies in the fact that you use a different specification in the models. In fit1 and fit2 you use the x and y combination, in fit3 the formula notation

If you switch all models in the formula notation (type ~ ., data = spam) you will see an Accuracy of 0.7135

If you switch all models in the x / y notation (spam, spam$type) you will see an Accuracy of 0.7266

There is probably someone who can explain why exactly this difference occurs. I have no idea except that it has something to do in the difference of how the S3 formula notation is processed versus the default notation of x and y.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ That's amazing, thank you so much for pointing that out. I had no idea that the notation alone could lead to such a difference. $\endgroup$ – msheffer Nov 12 '15 at 8:25
  • $\begingroup$ I read this a couple months ago while trying to figure out the same thing, but I think I just figured out the "why"! Looking at the docs for model.matrix and the formula object, it seems that these internally convert any factors to numerics using dummy variables so the formula can fit a linear model to the data. This is great for models such as SVM and kNN that require numerical input, but other classification models might want the factors as is so they can process them in other ways. In particular for Naive Bayes, which mathematically requires conditional independence between attributes... $\endgroup$ – Alecg_O Apr 10 '19 at 17:06
  • $\begingroup$ ..., the better way is to just treat the factors as numeric since it doesn't care about ordinality anyway. Converting the input to dummy variables actually hurts the model somewhat because they they will have a strong inverse correlation with each other and thus violate the conditional independence assumption. $\endgroup$ – Alecg_O Apr 10 '19 at 17:08
4
$\begingroup$

From my knowledge, for the first two models, you shouldn't be giving the whole spam data frame as the training variables (the class labels are considered as a feature in this case). Instead, you should use:

fit1 <- naiveBayes(spam[,-58], spam$type, type="raw")

this way it will produce the same results with (type ~., data=spam).

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

The difference is whether x includes the full matrix including class variable, or whether y is specifically excluded from x.

# y included in x -> Accuracy : 0.7266

fit3 <- train(
   x=spam,
   y=spam$type,
   method = "nb",
   trControl = trainControl(method="none"),
   tuneGrid = data.frame(usekernel=FALSE,fL=0,adjust=FALSE))

pred3 <- predict(fit3, spam, type="raw")
#Warnings that probability is 0 for some cases
confusionMatrix(pred3, spam$type)
#Accuracy : 0.7266

is different to:

# y excluded from x -> #Accuracy : 0.7135

fit3 <- train(
   x=spam[,-dim(spam)[2]],
   y=spam$type,
   method = "nb",
   trControl = trainControl(method="none"),
   tuneGrid = data.frame(usekernel=FALSE,fL=0,adjust=FALSE))

pred3 <- predict(fit3, spam, type="raw")
#Warnings that probability is 0 for some cases
confusionMatrix(pred3, spam$type)
#Accuracy : 0.7135
| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.