# Training and validating a naive gaussian bayes classifier

I've been reading up on the theory of naive bayes classifiers, specifically the ones in which the probability $p(x|c)$ is a random vector. The problem which I'm trying to solve takes a 4 dimensional vector and tries to classify it into 3 classes. It is known that out of the given 15000 samples the first 5000 belong to class 1, the next 5000 to class 2 and the last third to class 3. It is clear to me that I need to arrive at a $\mu$ and $\sigma$ for each $x_i$ in the feature vector $X$. However, I do not see how to use the resulting parameters in the classification script to classify an incoming sample from the test set.

I hope I defined the problem clearly and look for any pointers that can help me

From what I've seen people choose whatever class has the highest likelihood given the (naive) estimates for $\mu$ and $\sigma$ under the independence assumptions. That is, they choose the class $k_i = {\underset {k}{\operatorname {arg\,max} }} \prod_{p=1}^4 N(x_{ip} | \hat \mu_{p k}, \hat \sigma_{p k})$; where $p=1, \ldots, 4$ indexes the features, and $k_i \in \{1, 2, 3\}$ are the class predictions.

Here's what I mean using R:

# Simulated data ----------------------------------------------------------
K <- 3 # Number of classes
P <- 4 # Dimension of x
n.obs <- 15000
class.vec.tr <- rep(1:K, each=n.obs/K)

# Create true values of mu randomly.  Each row of mu.tr has the true mean vector
# for that class.
mu.tr <- matrix(nrow=K, ncol=P)
sd.tr <- matrix(nrow=K, ncol=P)
for(k in 1:K) {
mu.tr[k,] <- rnorm(P, sd=1.5)
sd.tr[k,] <- rgamma(P, 3, 1)
}

# Randomly generate the data
X.train <- matrix(nrow=n.obs, ncol=P)
X.test <- matrix(nrow=n.obs, ncol=P)
for(i in 1:n.obs) {
class.idx <- class.vec.tr[i]
mu.class <- mu.tr[class.idx,]
sd.class <- sd.tr[class.idx,]
X.train[i,] <- rnorm(P, mean=mu.class, sd=sd.class)
X.test[i,] <- rnorm(P, mean=mu.class, sd=sd.class)
}

# "Train" the model -------------------------------------------------------
# Calculate the mean within each class using the training data
mu.hat <- t(simplify2array(by(X.train, factor(class.vec.tr), colMeans, simplify=TRUE)))
# Calculate the standard deviation within each class using the training data,
# assuming feature independence
sd.hat <- t(simplify2array(by(X.train, factor(class.vec.tr), function(X.class) apply(X.class, 2, sd))))

# Evaluate performance on training data -----------------------------------
predicted.classes <- vector(length=n.obs)
for(j in 1:n.obs) {
X.observation <- X.test[j,]
x.dens <- vector(length=K)
for(k in 1:K) {
mu.hat.class <- mu.hat[k,] # Estimated mean for class k
sd.hat.class <- sd.hat[k,] # Estimated sd's for class k's features
x.dens[k] <- prod(dnorm(X.observation, mean=mu.hat.class, sd=sd.hat.class))
}
predicted.classes[j] <- which(x.dens==max(x.dens))
}
# See how we did:
table(predicted.classes, class.vec.tr)


Results in:

> table(predicted.classes, class.vec.tr)
class.vec.tr
predicted.classes    1    2    3
1 3843  545  642
2  318 3168  799
3  839 1287 3559
>

• So if I understand correctly, during testing you obtain a gaussian distribution, using the $\mu$ and $\sigma$ obtained from the testing phase. Am I correct? – Ankit Kulshrestha Feb 6 '17 at 1:22
• I think you meant "using the $\mu$ and $\sigma$ estimates obtained from the training phase". If so, then yes. You get estimates for $\mu$ and $\sigma$ during training, and then evaluate the likelihood on the test data to make predictions. I also probably should have mentioned in my answer that sometimes people put priors in for the class distributions. In my answer I assumed all three classes were equally likely a-priori. – AtALoss Feb 6 '17 at 2:13
• Oh yes, I meant training. Thanks for providing a very helpful code to understand as well – Ankit Kulshrestha Feb 6 '17 at 2:15