I'm trying to understand the something written in this paper.

At the bottom of page 7:

This means that if the naive Bayes model $$ p(y,\mathbf{x}) = p(y) \prod_k p(x_k|y) $$ is trained to maximize the conditional likelihood, we recover the same classifier as from logistic regression. Conversely, if the logistic regression model $$p(y|\mathbf{x}) = \frac{1}{Z(\mathbf{x})} exp\left\{\sum_k \lambda_k f_k(y,\mathbf{x}) \right\} $$ is interpreted generatively, as in $$p(y,\mathbf{x}) = \frac{exp\left\{\sum_k \lambda_k f_k(y,\mathbf{x}) \right\}}{\sum_{y', \mathbf{x}'}exp\left\{\sum_k \lambda_k f_k(y',\mathbf{x}') \right\}}$$ and is trained to maximize the joint likelihood p(y, x), then we recover the same classifier as from naive Bayes.

I'm having trouble seeing why this is.The next page goes on to say

...consider the family of naive Bayes distributions. This is a family of joint distributions whose conditionals all take the "logistic regression form". But there are many other joint models, some with complex dependencies among $\mathbf{x}$, whose conditional distributions also have the logistic regression form.

Aren't these statements contradictory? The conditioned-Bayes distributions are a subset of the logistic-regression-form distributions, so isn't it possible that maximizing over the latter yields a distribution not contained in the former?


1 Answer 1


The $f_k(y,{\bf x})$ introduced at the top of page 5 are of a very special form that, crucially, is linear in ${\bf x}$ and so (dropping the $k$ subscript for a moment) can be written $f(y,{\bf x}) = \alpha_1^{(y)}x_1 + \ldots + \alpha_K^{(y)}x_K$ . Exponentiating, that's just $\prod_k g(x_k|y)$, i.e. of Naïve-Bayes form. Your second equation is the paper's (1.7), which is just the more transparent (1.6) in disguise.

You are quite right that if $f_k(y,{\bf x})$ were not of this special form then the mapping to NB would not be guaranteed.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.