Derivation of Logistic regression

Question

I am trying to understand the derivation of logistic regression. Some books like the famous Elements of Statistical Learning just give you the formula without too much effort. Others try to explains it a little further.

The book A first course in Machine Learning for example does the following steps. First of all it starts from applying Bayes to

$$P(\mathbf{w|t,X)}=\frac{P(\mathbf{t|w,X)}P(\mathbf{w)}}{P(\mathbf{t|X})}$$

where $\mathbf{w}$ is the vector of the model parameters, $\mathbf{X}$ is the input matrix and $\mathbf{t}$ the output vector.

1. My first question is: Why is it starting from this? Should not it start from $P(\mathbf{t|w,X)}?$

Then it explains each term in the Bayes expression. Basically it says the numerator can be calculated while the denominator can not. The denominator is re-expressed as

$$P(\mathbf{t|X}) = \int P(\mathbf{t|w,X}) P(\mathbf{w}) d\mathbf{w}$$ and it says

[...] we cannot analytically perform the integration required to compute the marginal likelihood. In other words we have a function $g(\mathbf{w,X,t}) = P(\mathbf{t|w,X}) P(\mathbf{w})$ which we know is proportional to the posterior, $P(\mathbf{w|t,X)}=Zg(\mathbf{w,X,t})$, but we do not know the constant of proportionality $Z^{-1}$.

2. Why it is not possible to perform the integration? Also assuming that when talking about proportionality it is rexpressing the posterior as:$P(\mathbf{w|t,X)}= \frac{P(\mathbf{t|w,X)}P(\mathbf{w)}}{\int P(\mathbf{t|w,X}) P(\mathbf{w}) d\mathbf{w}}$, which is not explained, it is not saying which type of proportionality is that (inversely/directly) is it? if so how?

Since we cannot compute these quantities it says we are left with 3 options (as I have understood to find $\mathbf{w}$):

1)Find the single value of $\mathbf{w}$ that corresponds to the highest value of posterior. As $g(\mathbf{w,X,t})$ is proportional to the posterior, a maximum of $g$ will correspond to a maximum of the posterior. $Z^{-1}$ is not a function of $w$.

2)Approximate $P(\mathbf{w|t,X})$ with some other density that we can compute analytically.

3) Sample directly from the posterior $P(\mathbf{w|t,X)}$ knowing only $g$

3. It says the first is not very Bayesian: Why? It gives the example Newton-Raphson method but other books use this method to fit the regression. Also if $$P(\mathbf{w|t,X)}= \frac{P(\mathbf{t|w,X)}P(\mathbf{w)}}{\int P(\mathbf{t|w,X}) P(\mathbf{w}) d\mathbf{w}}$$ is valid, why a maximum in $g$ should correspond to a maximum in the posterior? And does logistic regression use a particular method or is it independent from that?

Then it says

Once we have the posterior density, we can predict the response class of new objects by taking an expectation with respect to this density: $P(t_{new}=1|x_{new}|\mathbf{X,t}) = \mathbf{E}_{P(\mathbf{w|t,X)}}{\frac{1}{1+e^{-\mathbf{w^Tx_{new}}}}}$

4. Why is the expectation taken over $P(\mathbf{w|t,X)}$?

5. What is the first motivation of logistic regression? Does it start from the desire to keep the linearity hence using $w^Tx$ but forcing the model to output probabilities so passing the linear function to a sigmoid? Or is this just a consequence of the desire to use log-odds?

Answer 1

1. My first question is: Why is it starting from this? Should not it start from $P(\mathbf{t|w,X)}?$

The statement is Bayes theorem. It is used to derive the posterior distribution from prior distribution $P(\mathbf{w)}$ and likelihood $P(\mathbf{t|w,X)}$. I don't know what do you mean by "starting" from the likelihood, but in Bayesian setting, you need both elements: priors and likelihood.

2. Why it is not possible to perform the integration?

It is not that the integral is "not possible to compute". There is no closed-form mathematical solution for this integral. In plain English, this means that there is no simple mathematical formula for it, where you could just plug-in the data and get the result. That is the reason, why we use methods like MCMC to approximate the integral.

Also assuming that when talking about proportionality it is rexpressing the posterior as: $P(\mathbf{w|t,X)}= \frac{P(\mathbf{t|w,X)}P(\mathbf{w)}}{\int P(\mathbf{t|w,X}) P(\mathbf{w}) d\mathbf{w}}$, which is not explained, it is not saying which type of proportionality is that (inversely/directly) is it? if so how?

Hard to say what you mean, but my best guess is that what was meant in the handbook is

$$ P(\mathbf{w|t,X)} \propto P(\mathbf{t|w,X)}P(\mathbf{w)} $$

so it is proportional if we drop the normalizing constant $Z^{-1}$ (using the notation mentioned by you later).

3. It says the first is not very Bayesian: Why? [...]

Because in Bayesian setting $\mathbf{w}$ is a random variable, so it has a distribution and you are interested in finding this distribution, rather then single "best" candidate.

[...] why a maximum in $g$ should correspond to a maximum in the posterior?

Because maximum of the posterior distribution is the maximum of the posterior distribution. As said earlier, $g$ is proportional to the posterior. In the optimization dividing $g$ by $Z$ would not make a difference in the result, so they are the same for this purpose. See also Normalizing constant in Bayes theorem, Normalizing constant irrelevant in Bayes theorem?, and Why Normalizing Factor is Required in Bayes Theorem?.

And does logistic regression use a particular method or is it independent from that?

The discussed model is a Bayesian logistic regression model. There is also the "classical" logistic regression that does not use priors and Bayes theorem at all, if this is what you are asking.

4. Why is the expectation taken over $P(\mathbf{w|t,X)}$?

Because that is what we are interested in, aren't we? We want to estimate the distribution of the parameters $\mathbf{w}$ given the data we observed $\mathbf{t,X}$. The procedure is about estimating the parameters of the model.

5. What is the first motivation of logistic regression? Does it start from the desire to keep the linearity hence using $w^Tx$ but forcing the model to output probabilities so passing the linear function to a sigmoid?

Yes, that's it. You can think of logistic regression as of linear regression that outputs probabilities, where the target variable is binary (or is a probability of binary event).