0
$\begingroup$

I have a question. In the following text, in the $\pi$ formula why $y$ appears? In a Bayesian framework, we model the parameter θ in the logistic equation as a random variable with a prior distribution $\pi_0$. Suppose that we observe a set of independent $\{(x_i,y_i)\}_{i=1}^n$. Define the vector $Y=(y_1,\cdots,y_n)\in \{0,1\}^n$ and let $X$ be the $n\times d$ matrix with $x_i$ as $i^{th}$-row. We choose the prior $\pi_0$ to be a Gaussian distribution with zero mean and covariance matrix proportional to the inverse of the sample covariance matrix $\sum_X=\frac{1}{n} X^TX$. Plugging in the formula for the prior and likelihood, we find that the posterior density is given by $\pi(\theta)=\pi(\theta|X,Y)\propto \mathrm{exp}\left\{ Y^TX\theta - \sum_{i=1}^n \log(1+e^{\theta^Tx_i})-\alpha||\sum_X^{1/2}\theta||_2^2\right\}$

It is assumed that the likelihood is logistic regression $p(y=1|x,\theta)=\frac{\exp(\theta^T x)}{1+\exp(\theta^T x)}$, then by taking logarithm we have $\log p(y=1|x,\theta) = \theta^T x+\log (1+\theta^T x)$. My question is that how $y$ appears in $Y^TX\theta$, where it comes from? why it does not appear in $\log (1+\theta^T x)$? $\mathrm{exp}\left\{ Y^TX\theta - \sum_{i=1}^n \log(1+e^{\theta^Tx_i})\right \}$?

https://arxiv.org/pdf/1801.02309.pdf section 4.3 page 21, second paragraph.

$\endgroup$
2
  • $\begingroup$ Please provide the reference. I see no objection to the logistic regression part. $\endgroup$ – Xi'an Jun 2 '20 at 15:17
  • $\begingroup$ I have added to the text: arxiv.org/pdf/1801.02309.pdf section 4.3 page 21, second paragraph. $\endgroup$ – Raz Jun 2 '20 at 18:46
2
$\begingroup$

From $$p(y=1|X,\theta)= \frac{\exp \theta^Tx}{1+\exp\theta^Tx}$$ (you forgot the $\exp$) and the corresponding $$p(y=0|X,\theta)= \frac{1}{1+\exp\theta^Tx}$$ the likelihood will be

$$L=\prod_i \left(\frac{\exp\theta^Tx_i}{1+\exp\theta^Tx_i}\right)^{y_i} \left(\frac{1}{1+\exp\theta^Tx_i}\right)^{(1-y_i)},$$ that is, $p(y=1|X,\theta)$ for observations with $y=1$ and $p(y=0|X,\theta)$ for observations with $y=0$.

The likelihood has to include $Y$, since $Y$ is what's random (or from a Bayesian viewpoint, it's what provides relative evidence about different values of $\theta$)

The likelihood for exponential family models such as the binomial can be written in the form $$L(\eta; y) \propto \exp\left(y\eta -A(\eta) \right)$$ For a canonical-link regression model you take $\eta=\theta^Tx$, and for logistic regression $\eta$ is the log odds: $\mathrm{logit}\, p(Y=1|X,\theta)$. That's the form used in the posterior density you quote.

Converting between the form in probabilities that I wrote first and the form in log-odds is slightly annoying but straightforward algebra.

$\endgroup$
2
  • $\begingroup$ Thank you so much. I would be thankful if you have a reference for this equation $$L(\eta; y) \propto \exp\left(y\eta -A(\eta) \right)$$ kindly refer me to that. $\endgroup$ – Raz Jun 3 '20 at 8:40
  • $\begingroup$ en.wikipedia.org/wiki/Exponential_family $\endgroup$ – whuber Jun 3 '20 at 13:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.