why $y$ appears in the log-likelihood of logistic regression?

Question

I have a question. In the following text, in the $\pi$ formula why $y$ appears? In a Bayesian framework, we model the parameter θ in the logistic equation as a random variable with a prior distribution $\pi_0$. Suppose that we observe a set of independent $\{(x_i,y_i)\}_{i=1}^n$. Define the vector $Y=(y_1,\cdots,y_n)\in \{0,1\}^n$ and let $X$ be the $n\times d$ matrix with $x_i$ as $i^{th}$-row. We choose the prior $\pi_0$ to be a Gaussian distribution with zero mean and covariance matrix proportional to the inverse of the sample covariance matrix $\sum_X=\frac{1}{n} X^TX$. Plugging in the formula for the prior and likelihood, we find that the posterior density is given by $\pi(\theta)=\pi(\theta|X,Y)\propto \mathrm{exp}\left\{ Y^TX\theta - \sum_{i=1}^n \log(1+e^{\theta^Tx_i})-\alpha||\sum_X^{1/2}\theta||_2^2\right\}$

It is assumed that the likelihood is logistic regression $p(y=1|x,\theta)=\frac{\exp(\theta^T x)}{1+\exp(\theta^T x)}$, then by taking logarithm we have $\log p(y=1|x,\theta) = \theta^T x+\log (1+\theta^T x)$. My question is that how $y$ appears in $Y^TX\theta$, where it comes from? why it does not appear in $\log (1+\theta^T x)$? $\mathrm{exp}\left\{ Y^TX\theta - \sum_{i=1}^n \log(1+e^{\theta^Tx_i})\right \}$?

https://arxiv.org/pdf/1801.02309.pdf section 4.3 page 21, second paragraph.

Answer 1

From $$p(y=1|X,\theta)= \frac{\exp \theta^Tx}{1+\exp\theta^Tx}$$ (you forgot the $\exp$) and the corresponding $$p(y=0|X,\theta)= \frac{1}{1+\exp\theta^Tx}$$ the likelihood will be

$$L=\prod_i \left(\frac{\exp\theta^Tx_i}{1+\exp\theta^Tx_i}\right)^{y_i} \left(\frac{1}{1+\exp\theta^Tx_i}\right)^{(1-y_i)},$$ that is, $p(y=1|X,\theta)$ for observations with $y=1$ and $p(y=0|X,\theta)$ for observations with $y=0$.

The likelihood has to include $Y$, since $Y$ is what's random (or from a Bayesian viewpoint, it's what provides relative evidence about different values of $\theta$)

The likelihood for exponential family models such as the binomial can be written in the form $$L(\eta; y) \propto \exp\left(y\eta -A(\eta) \right)$$ For a canonical-link regression model you take $\eta=\theta^Tx$, and for logistic regression $\eta$ is the log odds: $\mathrm{logit}\, p(Y=1|X,\theta)$. That's the form used in the posterior density you quote.

Converting between the form in probabilities that I wrote first and the form in log-odds is slightly annoying but straightforward algebra.