I try to implement Gaussian naive bayesian manually in R. I test my model on iris data set. I would like to build a predictive model. That is, I would like to consider, for each point, the correct class.

My problem:

I see that the posteriors for some points are larger than 1, which is wrong.

What I have tried:

  • I have divided the data into training and test datasets.

  • Then I assume equally prior for each class (0.3) (as the prior must sum to one).

  • I applied the Bayesian rule for the training data set. For this I do the following:

  • to compute the likelihood for each class, I computed the density (for each point in each class). den[[1]] is the density of the Sepal.Length. den[2] is the density of Sepal.Width. den[[3]] and den[[4]] are the density for the resining variables.

  • Then to compute the posterior for each point, I computed prior*likelihood, for each point.

However, I found that some posteriors values are larger than one. I believe I did something wrong, but do not know what is my mistake?

Any help, please?

Here is my code

##split the data based on its class
NewData <- split(iris, iris$Species)
## divide the data based on their class
NewDatSetosa <- NewData$setosa[,1:4]
NewDatVersicolor <- NewData$versicolor[,1:4]
NewDatVirginica <- NewData$virginica[,1:4]

##divide the data into train and test datasets
datadivision <- sample(2, nrow(iris), replace=TRUE, prob=c(0.67, 0.33))
iris.training <- iris[datadivision==1, 1:4]
iris.test <- iris[datadivision==2, 1:4]
##Compute the posterior for each class

posterior <- function(data,prior,newDat){
  post <- list()
##compute the density for each point in the training data set
  den <- lapply(1:4, function(i) dnorm(data[[i]], mean(newDat[[i]]), sd(newDat[[i]])))
## compute the posterior for each point. den[[1]] is the density of the 
##Sepal.Length. den[2] is the density of Sepal.Width. den[3] and den[4] ##are the density for the resining variables.

     post <- prior*den[[1]]*den[[2]]*den[[3]]*den[[4]]
post.Setosa <- posterior(iris.training, prior=0.3, NewDatSetosa)
post.Versicolor <- posterior(iris.training, prior=0.3, NewDatVersicolor)
post.Virginica <- posterior(iris.training, prior=0.3, NewDatVirginica)
  • $\begingroup$ Not sure I understand what you are doing, but why would a normal distribution (not restricted to [0, 1]) for the likelihood make sense, if you expect an posterior to have something to do with a probability of class membership? $\endgroup$ – Björn Apr 30 '19 at 11:35
  • $\begingroup$ @Björn As I understand the normal distribution must be estimated for each class. I tried to do a classification problem. So, I estimate the normal distribution for each class and then apply the Bayesian rule. $\endgroup$ – Mary Apr 30 '19 at 11:37

Your priors should be $1/3$, not 0.3. But, this is the minor issue. It's true that your posterior, i.e. $p(c|x)$, where $c$ represents the class can't be larger than $1$. However, it's not equal to prior * likelihood; rather it is proportional to it: $$p(c|x)=\frac{p(x|c)p(c)}{p(x)}\propto p(x|c)p(c)$$ So, what you've calculated is not a valid PDF or PMF. It should be normalized by $p(x)=\sum_{c_i} p(x|c_i)p(c_i)$. But, you can do the classification w/o the normalization step because $p(x)$ doesn't depend on specific $c_i$.

  • $\begingroup$ Thank you so much for your help. Hence, I need to divide prior*den[[1]]*den[[2]]*den[[3]]*den[[4]] by the their sum. $\endgroup$ – Mary Apr 30 '19 at 11:35
  • 1
    $\begingroup$ @Mary , yes sum wrt each class. $\endgroup$ – gunes Apr 30 '19 at 13:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.