4
$\begingroup$

I am trying to understand the analogy between a single layer neural network and naive Bayes classifier. Particularly, I want to know if, in a neural network, the variables are independent given the class and if the bias term represents the prior probability. (As far as I understand, if I take the log of the naive bayes expression, it seems to give a term similar to the weighted sum +bias in neural network as worked out here wiki )

Moreover, if single layer neural network have exactly same representative power as naive Bayes (i.e. the features are independent given the class), I want to understand, how multilayer neural network captures the dependence among features.

$\endgroup$
3
$\begingroup$

A single layer neural network with sigmoidal or softmax outputs and cross entropy loss is equivalent to logistic regression (or multinomial logistic regression). Given that, the following chapter may be of interest:

Mitchell (2015). Generative and Discriminative Classifiers: Naive Bayes and Logistic Regression.

He shows that the conditional distribution (probability of class given input) of a Gaussian naive Bayes classifier can be written exactly in the form of logistic regression, with a particular choice of the weights and biases. So, your intuition that there's a connection is spot on. The bias term in this form depends on the prior class probabilities (as you suggest), but also on the means and variances.

For a Gaussian naive Bayes classifier, let:

  • $Y$ be the class (0 or 1)
  • $X = (X_1, ..., X_n)$ be continuous-valued input variables/attributes
  • $\pi$ be the prior class probability $p(Y = 1)$
  • The inputs $X_i$ be conditionally independent (given $Y$). The conditional distribution $p(X_i \mid Y = y_k)$ is Gaussian, with the form $N(\mu_{ik}, \sigma_i)$

Mitchell (equation 22) gives the conditional probability of class given input as:

$$p(Y=1 \mid x) = \frac{1}{1 + \exp \left ( w_0 + \sum_{i=1}^{n} w_i X_i \right )}$$

Where:

$$w_i = \frac{\mu_{i0} - \mu_{i1}}{\sigma_i^2}$$

$$w_0 = \ln \frac{1 - \pi}{\pi} + \sum_i \frac{\mu_{i1}^2 - \mu_{i0}^2}{2 \sigma_i^2}$$

This is exactly the form of logistic regression, where $w_i$ are the weights and $w_0$ is the bias.

However, logistic regression (and therefore single-layer neural nets) don't necessarily impose these forms on the weights/bias, and are therefore more general than naive Bayes.

Regarding your question about conditional independence...Naive Bayes is a generative model, so it must explicitly specify a distribution for the inputs. It imposes the conditional independence assumption. But, single-layer neural nets and logistic regression are discriminative models; they only model the probability of the class given the input. The state of units in the input layer is completely determined by the data, so whether or not they're conditionally independent (given the class) is determined by the data, not the model.

For a binary classification problem, we could imagine two point clouds (one for each class). A Gaussian naive Bayes classifier would model the clouds as Gaussian distributions. Because of the conditional independence assumption, the axes of the Gaussians would be parallel to the data axes. Single layer neural nets/logistic regression would project all points onto a line, and model the class probability as a sigmoidal function of position along the line. The shape of the point clouds wouldn't be explicitly taken into account, and could be anything (non-axis-parallel Gaussians, banana-shaped, stars, etc.).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.