Accuracy rate in naive Bayes classification

Question

I am trying to use a naive Bayes classification technique to predict fraudsters (Caller). My training set of 138 instances has 5 columns viz. Morning, Afternoon, Evening, Night and Caller. Morning has 8 names; the rest all have 3. The names are unique to each column.

> levels(mlcallers$Morning)
    [1] "Kelly"   "Larry"   "Mark"    "Nancy"   "Olga"    "Peter"   "Quentin" "Robert" 
    > levels(mlcallers$Afternoon)
[1] "George" "Harry"  "John"  
> levels(mlcallers$Evening)
    [1] "David" "Emily" "Frank"
    > levels(mlcallers$Night)
[1] "Alex"  "Beth"  "Clark"
> levels(mlcallers$Caller)
[1] "Sally"    "Vince"    "Virginia"

## Training data set
> summary(mlcallers)
Morning    Afternoon   Evening     Night         Caller  
 Olga   :29   George:31   David:35   Alex :43   Sally   :47  
 Peter  :29   Harry :49   Emily:44   Beth :52   Vince   :43  
 Quentin:22   John  :58   Frank:59   Clark:43   Virginia:48  
 Robert :13                                                  
 Mark   :12                                                  
 Nancy  :12                                                  
 (Other):21  

## Test data set
> summary(testdata) 
Morning   Afternoon  Evening    Night  
 Kelly  :2   George:7   David:7   Alex :6  
 Larry  :1   Harry :3   Emily:1   Beth :2  
 Mark   :1   John  :5   Frank:7   Clark:7  
 Nancy  :1                                 
 Olga   :1                                 
 Quentin:4                                 
 Robert :5 

I need to predict the Caller (probable fraudster) in the test data set and also report their confidence. My attempt is as follows:

> library(e1071)
> model <- naiveBayes(Caller~., data=mlcallers)
> predict(model, testdata)
[1] Sally    Sally    Sally    Vince    Sally    Vince    Vince    Virginia    Virginia     Virginia    Virginia    Sally   
[13] Sally    Virginia Sally   
Levels: Sally Vince Virginia
> predict(model, testdata, type="raw")
       Sally       Vince    Virginia
 [1,] 0.81806260 0.155576135 0.026361264
 [2,] 0.93405930 0.057871797 0.008068906
 [3,] 0.82235738 0.122064542 0.055578078
 [4,] 0.40028059 0.540385630 0.059333784
 [5,] 0.74235622 0.068005993 0.189637784
 [6,] 0.14954988 0.723948800 0.126501321
 [7,] 0.12730200 0.657333452 0.215364548
 [8,] 0.12960601 0.001299552 0.869094437
 [9,] 0.02378972 0.001962829 0.974247449
[10,] 0.01420016 0.171655799 0.814144039
[11,] 0.01588595 0.101070603 0.883043443
[12,] 0.82235738 0.122064542 0.055578078
[13,] 0.93569932 0.052176068 0.012124610
[14,] 0.02152574 0.361402108 0.617072156
[15,] 0.74235622 0.068005993 0.189637784

Since the test data has only 15 instances, to find out the accuracy, I have used:

> pred <- predict(model, newdata=testdata, laplace=3)
> table(pred, mlcallers[65:79,5])

pred       Sally Vince Virginia
  Sally        1     4        2
  Vince        0     3        0
  Virginia     0     0        5

....which gives the highest accuracy in some random trials.

My question has 2 parts:

1. Is this the correct approach to the problem?
2. Is there any way I can find out the highest accuracy without me having to randomly select 15 rows from the training data (different random selection like [110:124], yields different accuracy results?

Answer 1

From what I understand, your question basically boils down to one about model selection and cross-validation.

When it comes to getting a prediction of test error, you can't hand-pick the best training/test sets in order to minimize your error. Before starting any sort of parameter tuning or model selection, you must separate your labeled data into a training and test set. Usually we see something such as 80/20 or 90/10 (sometimes even 50/50 for huge datasets) for test/train respectively.

In your case, you want to make as much use of your limited data as possible so I suggest reading up on cross-validation.

You would want to use cross-validation to select the best model (and tune the parameters within the Naive Bayes, if that's what you are set on using) by calculating the error within each fold. Once you 'know' what model will work best on your data using your test/training splits, you would train your final production model on the full data.

P.S. - I would suggest looking into a Random Forest model for this sort of data just because in my experience it tends to work better most of the time.