Naive Bayes is a "special case" of logistic regression - which other models?

Question

Suppose $Y \in \{0, 1\}$ is a response variable and $X = (X_1, \cdots, X_p)$ are covariates with $X_j \in \{0, 1\}$ for each $j = 1, \cdots, p$.

In the Naive Bayes model, we assume conditional independence of the covariates given the class label, so that

$$p_{\theta}(y = 1 \mid x) = \frac{\prod_{j=1}^p p_{\theta}(x_j \mid y) p_\theta(y)}{p(x)}.$$

If we denote the parameters as $\theta_0 := P(Y = 1)$ and $$\theta_{j,k} = P(X_j = 1 \mid Y = k),\ 1 \leq j \leq p, k \in \{0, 1\}$$

then we can write $$\begin{align*} p_{\theta}(y = 1 \mid x) &= \frac{\prod_{j=1}^p \theta_{j, 1}^{x_j} (1 - \theta_{j, 1})^{1 - x_j} \theta_0}{ \prod_{j=1}^p \theta_{j, 1}^{x_j} (1 - \theta_{j, 1})^{1 - x_j} \theta_0 + \prod_{j=1}^p \theta_{j, 0}^{x_j} (1 - \theta_{j, 0})^{1 - x_j} (1 - \theta_0) } \\ &= \frac{1}{1 + \prod_{j=1}^p \left(\frac{\theta_{j,0}}{\theta_{j,1}}\right)^{x_{j}} \left(\frac{1 - \theta_{j,0}}{1 - \theta_{j, 1}} \right)^{1 - x_j} \frac{1 - \theta_0}{\theta_0}} \\ &= \frac{1}{1 + \exp \log \prod_{j=1}^p \left(\frac{\theta_{j,0}}{\theta_{j,1}}\right)^{x_{j}} \left(\frac{1 - \theta_{j,0}}{1 - \theta_{j, 1}} \right)^{1 - x_j} \frac{1 - \theta_0}{\theta_0}} \\ &= \frac{1}{1 + \exp(-(\beta_0 + \beta_1 x_1 + \cdots + \beta_p x_p))} \end{align*}$$

where $$\beta_0 = \log(\frac{1 - \theta_)}{\theta_0}) + \sum_{j=1}^p \log(\frac{1 - \theta_{j,0}}{1 - \theta_{j,1}})$$ and $$\beta_j = \log(\frac{\theta_{j,0}}{\theta_{j,1}}) - \log(\frac{1 - \theta_{j,0}}{1 - \theta_{j,1}}).$$

So, in a sense, Naive Bayes is a special case of logistic regression. The difference is that by using a generative model, we are able to make specific assumptions about the form of the likelihood $p_{\theta}(x \mid y)$ and the prior $p_{\theta}(y)$. A similar derivation can be shown for GDA (Gaussian Discriminant Analysis), where $p_{\theta}(x \mid y)$ is normally distributed.

My question is, which other generative models $p_{\theta}(x, y)$ where $p_{\theta}(x \mid y)$ is an exponential family, can be thought of as being a "special case" of logistic regression, in this way? For what other distributions is a derivation like this possible?

Answer 1

We can write $$\begin{align*} p(y = 1 \mid x) &= \frac{p(x \mid y = 1)p(y = 1)}{p(x \mid y = 1)p(y = 1) + p(x \mid y = 0)p(y = 0)} \\ &= \frac{1}{1 + \frac{p(x \mid y = 0)p(y = 0)}{p(x \mid y = 1)p(y = 1)}} \\ &= \frac{1}{1 + \exp \log \left\{ \frac{p(x \mid y = 0)p(y = 0)}{p(x \mid y = 1)p(y = 1)} \right\}}. \end{align*}$$

We want $\log \left\{ \frac{p(x \mid y = 0)p(y = 0)}{p(x \mid y = 1)p(y = 1)} \right\}$ to be an affine function of $x$. Indeed this is true if $p(x \mid y)$ is in the exponential family, since if

$$p(x \mid y) = h(x) \exp\{\eta(y)^T x - a(\eta(y))\}$$

Then

$$\log \left\{ \frac{p(x \mid y = 0)p(y = 0)}{p(x \mid y = 1)p(y = 1)} \right\} = (\eta(0) - \eta(1))^T x - (a(\eta(0)) + a(\eta(1))).$$

So

$$p(y = 1 \mid x) = \frac{1}{1 + \exp(\beta_0 + \beta_1 x_1 + \cdots + \beta_p x_p)}$$

where $$\beta_0 = \log\{\frac{p(y=0)}{p(y=1)}\} - (a(\eta(0)) + a(\eta(1)))$$

and $$\beta_j = (\eta(0) - \eta(1))_j.$$