I've been reading up on the theory of naive bayes classifiers, specifically the ones in which the probability $p(x|c)$ is a random vector. The problem which I'm trying to solve takes a 4 dimensional vector and tries to classify it into 3 classes. It is known that out of the given 15000 samples the first 5000 belong to class 1, the next 5000 to class 2 and the last third to class 3. It is clear to me that I need to arrive at a $\mu$ and $\sigma$ for each $x_i$ in the feature vector $X$. However, I do not see how to use the resulting parameters in the classification script to classify an incoming sample from the test set.

I hope I defined the problem clearly and look for any pointers that can help me


1 Answer 1


From what I've seen people choose whatever class has the highest likelihood given the (naive) estimates for $\mu$ and $\sigma$ under the independence assumptions. That is, they choose the class $k_i = {\underset {k}{\operatorname {arg\,max} }} \prod_{p=1}^4 N(x_{ip} | \hat \mu_{p k}, \hat \sigma_{p k})$; where $p=1, \ldots, 4$ indexes the features, and $k_i \in \{1, 2, 3\}$ are the class predictions.

Here's what I mean using R:

# Simulated data ----------------------------------------------------------
K <- 3 # Number of classes
P <- 4 # Dimension of x
n.obs <- 15000
class.vec.tr <- rep(1:K, each=n.obs/K)

# Create true values of mu randomly.  Each row of mu.tr has the true mean vector
# for that class.
mu.tr <- matrix(nrow=K, ncol=P)
sd.tr <- matrix(nrow=K, ncol=P)
for(k in 1:K) {
    mu.tr[k,] <- rnorm(P, sd=1.5)
    sd.tr[k,] <- rgamma(P, 3, 1)

# Randomly generate the data
X.train <- matrix(nrow=n.obs, ncol=P)
X.test <- matrix(nrow=n.obs, ncol=P)
for(i in 1:n.obs) {
    class.idx <- class.vec.tr[i]
    mu.class <- mu.tr[class.idx,]
    sd.class <- sd.tr[class.idx,]
    X.train[i,] <- rnorm(P, mean=mu.class, sd=sd.class)
    X.test[i,] <- rnorm(P, mean=mu.class, sd=sd.class)

# "Train" the model -------------------------------------------------------
# Calculate the mean within each class using the training data
mu.hat <- t(simplify2array(by(X.train, factor(class.vec.tr), colMeans, simplify=TRUE)))
# Calculate the standard deviation within each class using the training data,
# assuming feature independence
sd.hat <- t(simplify2array(by(X.train, factor(class.vec.tr), function(X.class) apply(X.class, 2, sd))))

# Evaluate performance on training data -----------------------------------
predicted.classes <- vector(length=n.obs)
for(j in 1:n.obs) {
    X.observation <- X.test[j,]
    x.dens <- vector(length=K)
    for(k in 1:K) {
        mu.hat.class <- mu.hat[k,] # Estimated mean for class k
        sd.hat.class <- sd.hat[k,] # Estimated sd's for class k's features
        x.dens[k] <- prod(dnorm(X.observation, mean=mu.hat.class, sd=sd.hat.class))
    predicted.classes[j] <- which(x.dens==max(x.dens))
# See how we did:
table(predicted.classes, class.vec.tr)

Results in:

> table(predicted.classes, class.vec.tr)
predicted.classes    1    2    3
                1 3843  545  642
                2  318 3168  799
                3  839 1287 3559
  • $\begingroup$ So if I understand correctly, during testing you obtain a gaussian distribution, using the $\mu$ and $\sigma$ obtained from the testing phase. Am I correct? $\endgroup$ Commented Feb 6, 2017 at 1:22
  • $\begingroup$ I think you meant "using the $\mu$ and $\sigma$ estimates obtained from the training phase". If so, then yes. You get estimates for $\mu$ and $\sigma$ during training, and then evaluate the likelihood on the test data to make predictions. I also probably should have mentioned in my answer that sometimes people put priors in for the class distributions. In my answer I assumed all three classes were equally likely a-priori. $\endgroup$
    – AtALoss
    Commented Feb 6, 2017 at 2:13
  • $\begingroup$ Oh yes, I meant training. Thanks for providing a very helpful code to understand as well $\endgroup$ Commented Feb 6, 2017 at 2:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.