Very poor accuracy in Naive Bayes for ancestry/surname classification

Question

Naive Bayes has a very good reputation on the classification of surnames by ancestry (see http://www.ncbi.nlm.nih.gov/pubmed/24944286).

I would like to apply a Naive Bayes classifier in R to identify the ancestry of individuals based on character 3-gram of their surnames.

My dataset has 5 classes: IBR, JPN, GER, EAS and ITA. Surprisingly, the code bellow classifies all surnames as Japanese or German ancestry; the accuracy is only 0.09 (these two classes are the least common in my sample!)

library(e1071)
library(caret)
test <- read.csv("https://dl.dropboxusercontent.com/u/116353/test.csv", header=T)
test <- test[,-1]
train <- read.csv("https://dl.dropboxusercontent.com/u/116353/train.csv", header=T)
train <- train[,-1]
model  <- naiveBayes(nation~., data=train, laplace=1) 
predictions <- predict(model, newdata=test[,-1])
confusionMatrix(predictions, test$nation)

• When I increase the size of the training set the problem get worse and everybody is classified as Japanese;

• I am pretty sure that the data is fine, because I tested other classifiers (SVM and Carvar & Trenkle)and they performed ok.

• When I increase the value of "threshold"=0.05, accuracy goes to 0.5.

  Thank you!

Answer 1

Maybe the problem is the binary predictors are defined as numeric variables with codes 0/1, and then, this function considers the variables are numeric and fits normal distributions for the condictional probabilities. Try to transform the variables as factors, then the function calculates the probabilities based in the frecuencies.

Please, run this code for understand what is the difference:

naiveBayes( nation ~ as.factor(acc) + as.factor(ach) , data = train )
naiveBayes( nation ~ acc+ ach , data = train )