# why $y$ appears in the log-likelihood of logistic regression?

I have a question. In the following text, in the $$\pi$$ formula why $$y$$ appears? In a Bayesian framework, we model the parameter θ in the logistic equation as a random variable with a prior distribution $$\pi_0$$. Suppose that we observe a set of independent $$\{(x_i,y_i)\}_{i=1}^n$$. Define the vector $$Y=(y_1,\cdots,y_n)\in \{0,1\}^n$$ and let $$X$$ be the $$n\times d$$ matrix with $$x_i$$ as $$i^{th}$$-row. We choose the prior $$\pi_0$$ to be a Gaussian distribution with zero mean and covariance matrix proportional to the inverse of the sample covariance matrix $$\sum_X=\frac{1}{n} X^TX$$. Plugging in the formula for the prior and likelihood, we find that the posterior density is given by $$\pi(\theta)=\pi(\theta|X,Y)\propto \mathrm{exp}\left\{ Y^TX\theta - \sum_{i=1}^n \log(1+e^{\theta^Tx_i})-\alpha||\sum_X^{1/2}\theta||_2^2\right\}$$

It is assumed that the likelihood is logistic regression $$p(y=1|x,\theta)=\frac{\exp(\theta^T x)}{1+\exp(\theta^T x)}$$, then by taking logarithm we have $$\log p(y=1|x,\theta) = \theta^T x+\log (1+\theta^T x)$$. My question is that how $$y$$ appears in $$Y^TX\theta$$, where it comes from? why it does not appear in $$\log (1+\theta^T x)$$? $$\mathrm{exp}\left\{ Y^TX\theta - \sum_{i=1}^n \log(1+e^{\theta^Tx_i})\right \}$$?

https://arxiv.org/pdf/1801.02309.pdf section 4.3 page 21, second paragraph.

• Please provide the reference. I see no objection to the logistic regression part. Jun 2, 2020 at 15:17
• I have added to the text: arxiv.org/pdf/1801.02309.pdf section 4.3 page 21, second paragraph.
– Raz
Jun 2, 2020 at 18:46

From $$p(y=1|X,\theta)= \frac{\exp \theta^Tx}{1+\exp\theta^Tx}$$ (you forgot the $$\exp$$) and the corresponding $$p(y=0|X,\theta)= \frac{1}{1+\exp\theta^Tx}$$ the likelihood will be

$$L=\prod_i \left(\frac{\exp\theta^Tx_i}{1+\exp\theta^Tx_i}\right)^{y_i} \left(\frac{1}{1+\exp\theta^Tx_i}\right)^{(1-y_i)},$$ that is, $$p(y=1|X,\theta)$$ for observations with $$y=1$$ and $$p(y=0|X,\theta)$$ for observations with $$y=0$$.

The likelihood has to include $$Y$$, since $$Y$$ is what's random (or from a Bayesian viewpoint, it's what provides relative evidence about different values of $$\theta$$)

The likelihood for exponential family models such as the binomial can be written in the form $$L(\eta; y) \propto \exp\left(y\eta -A(\eta) \right)$$ For a canonical-link regression model you take $$\eta=\theta^Tx$$, and for logistic regression $$\eta$$ is the log odds: $$\mathrm{logit}\, p(Y=1|X,\theta)$$. That's the form used in the posterior density you quote.

Converting between the form in probabilities that I wrote first and the form in log-odds is slightly annoying but straightforward algebra.

• Thank you so much. I would be thankful if you have a reference for this equation $$L(\eta; y) \propto \exp\left(y\eta -A(\eta) \right)$$ kindly refer me to that.
– Raz
Jun 3, 2020 at 8:40
• en.wikipedia.org/wiki/Exponential_family
– whuber
Jun 3, 2020 at 13:51