1
$\begingroup$

Suppose $Y \in \{0, 1\}$ is a response variable and $X = (X_1, \cdots, X_p)$ are covariates with $X_j \in \{0, 1\}$ for each $j = 1, \cdots, p$.

In the Naive Bayes model, we assume conditional independence of the covariates given the class label, so that

$$p_{\theta}(y = 1 \mid x) = \frac{\prod_{j=1}^p p_{\theta}(x_j \mid y) p_\theta(y)}{p(x)}.$$

If we denote the parameters as $\theta_0 := P(Y = 1)$ and $$\theta_{j,k} = P(X_j = 1 \mid Y = k),\ 1 \leq j \leq p, k \in \{0, 1\}$$

then we can write $$\begin{align*} p_{\theta}(y = 1 \mid x) &= \frac{\prod_{j=1}^p \theta_{j, 1}^{x_j} (1 - \theta_{j, 1})^{1 - x_j} \theta_0}{ \prod_{j=1}^p \theta_{j, 1}^{x_j} (1 - \theta_{j, 1})^{1 - x_j} \theta_0 + \prod_{j=1}^p \theta_{j, 0}^{x_j} (1 - \theta_{j, 0})^{1 - x_j} (1 - \theta_0) } \\ &= \frac{1}{1 + \prod_{j=1}^p \left(\frac{\theta_{j,0}}{\theta_{j,1}}\right)^{x_{j}} \left(\frac{1 - \theta_{j,0}}{1 - \theta_{j, 1}} \right)^{1 - x_j} \frac{1 - \theta_0}{\theta_0}} \\ &= \frac{1}{1 + \exp \log \prod_{j=1}^p \left(\frac{\theta_{j,0}}{\theta_{j,1}}\right)^{x_{j}} \left(\frac{1 - \theta_{j,0}}{1 - \theta_{j, 1}} \right)^{1 - x_j} \frac{1 - \theta_0}{\theta_0}} \\ &= \frac{1}{1 + \exp(-(\beta_0 + \beta_1 x_1 + \cdots + \beta_p x_p))} \end{align*}$$

where $$\beta_0 = \log(\frac{1 - \theta_)}{\theta_0}) + \sum_{j=1}^p \log(\frac{1 - \theta_{j,0}}{1 - \theta_{j,1}})$$ and $$\beta_j = \log(\frac{\theta_{j,0}}{\theta_{j,1}}) - \log(\frac{1 - \theta_{j,0}}{1 - \theta_{j,1}}).$$

So, in a sense, Naive Bayes is a special case of logistic regression. The difference is that by using a generative model, we are able to make specific assumptions about the form of the likelihood $p_{\theta}(x \mid y)$ and the prior $p_{\theta}(y)$. A similar derivation can be shown for GDA (Gaussian Discriminant Analysis), where $p_{\theta}(x \mid y)$ is normally distributed.

My question is, which other generative models $p_{\theta}(x, y)$ where $p_{\theta}(x \mid y)$ is an exponential family, can be thought of as being a "special case" of logistic regression, in this way? For what other distributions is a derivation like this possible?

$\endgroup$

1 Answer 1

0
$\begingroup$

We can write $$\begin{align*} p(y = 1 \mid x) &= \frac{p(x \mid y = 1)p(y = 1)}{p(x \mid y = 1)p(y = 1) + p(x \mid y = 0)p(y = 0)} \\ &= \frac{1}{1 + \frac{p(x \mid y = 0)p(y = 0)}{p(x \mid y = 1)p(y = 1)}} \\ &= \frac{1}{1 + \exp \log \left\{ \frac{p(x \mid y = 0)p(y = 0)}{p(x \mid y = 1)p(y = 1)} \right\}}. \end{align*}$$

We want $\log \left\{ \frac{p(x \mid y = 0)p(y = 0)}{p(x \mid y = 1)p(y = 1)} \right\}$ to be an affine function of $x$. Indeed this is true if $p(x \mid y)$ is in the exponential family, since if

$$p(x \mid y) = h(x) \exp\{\eta(y)^T x - a(\eta(y))\}$$

Then

$$\log \left\{ \frac{p(x \mid y = 0)p(y = 0)}{p(x \mid y = 1)p(y = 1)} \right\} = (\eta(0) - \eta(1))^T x - (a(\eta(0)) + a(\eta(1))).$$

So

$$p(y = 1 \mid x) = \frac{1}{1 + \exp(\beta_0 + \beta_1 x_1 + \cdots + \beta_p x_p)}$$

where $$\beta_0 = \log\{\frac{p(y=0)}{p(y=1)}\} - (a(\eta(0)) + a(\eta(1)))$$

and $$\beta_j = (\eta(0) - \eta(1))_j.$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.