Naive Bayes on continuous variables

Question

Please allow me to ask a basic question. I understand the mechanics of Naive Bayes for discrete variables, and can redo the calculations "by hand". (code of HouseVotes84 all the way per below).

However - I am struggling to see how the mechanics work for continuous variables (example code per below). How does the package calculate the conditional probabilities [, 1] and [, 2] in the table per below? As any individual X value is unique, does it create a range around each point, and calculate relative frequencies within these ranges (e.g. if the point is +0.311, does it evaluate the incidence of blue and orange spots in e.g. a range of 0.1 and +0.5?) This might be basic question - apologies if so.

Table

A-priori probabilities:
Y
  blue orange 
   0.5    0.5 

Conditional probabilities:
        values
Y              [,1]      [,2]
  blue   0.08703793 0.9238799
  orange 1.33486433 0.9988389

Code

blue=rep("blue",50); orange=rep("orange",50); colour=c(blue,orange); values1=rnorm(50,0,1); values2=rnorm(50,1,1); values=c(values1,values2)
df=data.frame(colour,values)

(model <- naiveBayes(colour ~ ., data = df))
(predict(model, df[1:10,]))
(predict(model, df[1:10,], type = "raw"))
(pred <- predict(model, df))
table(pred, df$colour)

## Categorical data only:
library(e1071)
data(HouseVotes84, package = "mlbench")
HouseVotes84=HouseVotes84[,1:3]
(model <- naiveBayes(Class ~ ., data = HouseVotes84))
(predict(model, HouseVotes84[1:10,]))
(predict(model, HouseVotes84[1:10,], type = "raw"))
(pred <- predict(model, HouseVotes84))
table(pred, HouseVotes84$Class)

Answer 1

From the R package (e1071) and the function naiveBayes that you're using:

The standard naive Bayes classifier (at least this implementation) assumes independence of the predictor variables, and Gaussian distribution (given the target class) of metric predictors. For attributes with missing values, the corresponding table entries are omitted for prediction.

It's pretty standard for continuous variables in a naive Bayes that a normal distribution is considered for these variables and a mean and standard deviation can then be calculated and then using some standard z-table calculations probabilities can be estimated for each of your continuous variables to make the naive Bayes classifier. I thought that it was possible to change the distributional assumption in this package, but apparently I'm wrong.

There is another R package (klaR) where you can change the density kernel. (the function is NaiveBayes). From the package:

NaiveBayes(x, grouping, prior, usekernel = FALSE, fL = 0, ...)

usekernel

if TRUE a kernel density estimate (density) is used for denstity estimation. If FALSE a normal density is estimated.

density(x, bw = "nrd0", adjust = 1,
        kernel = c("gaussian", "epanechnikov", "rectangular",
                   "triangular", "biweight",
                   "cosine", "optcosine")

Answer 2

I was working on a project not that long ago for which I needed to compute a naive bayes classifier (from scratch). I started out in R, but once I had the process down, I moved the code to Python. Here's my code that I began with. Don't expect it to be polished. For the most part, I followed Wikipedia's example (https://en.wikipedia.org/wiki/Naive_Bayes_classifier#Examples).

The steps are simple:

1. calculate the a priori probabilities which are the proportion of classes

2. For your continuous data, assume a normal distribution and calculate the mean and standard deviation.

3. To classify observations, take the new input x, calculate dnorm(x, mu, sigma) where mu and sigma come from step 2.

4. Sum up log(apriori) + log(dnorm(...)).

At this point, log(dnorm(...)) should contain two log-values (in my example). The probability that they belong in class 0 and probability they belong in class 1. This is the point Eric Peterson makes in his second paragraph.

5. Calculate the posterior probabilities

I also compared my results to R library e1071. My probability results do not line up with theirs for this simple case, though the classification does. In their predict.naiveBayes function, they have something like log(apriori) + apply(log(sapply(...compute dnorm code here...)), 1, sum) which returns log(apriori) + log(1) = log(apriori) which is an error so their classification is solely based on the a priori probabilities (actually, they use counts not the probabilities).

Anyways, I hope this helps you (and anyone else) see what's under the hood as it was not clear to me either.

n=30 ## samples
set.seed(123)
x = c(rnorm(n/2, 10, 2), rnorm(n/2, 0, 2))
y = as.factor(c(rep(0, 20), rep(1, 10)))
y


#library(e1071)
#nb = naiveBayes(x, y, laplace = 0)
#nb

#nb_predictions = predict(nb, x[1], type='raw')
#nb_predictions



library(dplyr)

nbc <- function(x, y){
  df <- as.data.frame(cbind(x,y))
  a_priori <- table(y) #/length(y)

  cond_probs <- df %>% group_by(y) %>% summarise(means = mean(x), var = sd(x))

  print("A Priori Probabilities")
  print(a_priori/sum(a_priori))

  print("conditional probabilities \n")
  print(cond_probs)

  return(list(apriori = a_priori, tables = cond_probs))
}



predict_nbc <- function(model, new_x){
  apriori = as.matrix(model$apriori)
  a = log(apriori/sum(apriori))
  msd = as.matrix(model$tables)[,c(2,3)] ## creates 3 columsn; first is junk
  probs = sapply(new_x, function(v) dnorm(x = v, mean = msd[,1], sd = msd[,2]))
  b = log(probs)
  #L = a + b ## works for 1 new obs
  L = apply(X = b, MARGIN = 2, FUN = function(v) a + v)

  results <- apply(X = L, MARGIN = 2, function(x){
                   sapply(x, function(lp){ 1/sum(exp(x - lp)) }) ## numerically stable
  })
  return(results)
}



fit = nbc(x,y)

fit ## my naive bayes classifier model

myres = predict_nbc(fit, new_x = x[1:4])
myres