# Derivation of Logistic regression

I am trying to understand the derivation of logistic regression. Some books like the famous Elements of Statistical Learning just give you the formula without too much effort. Others try to explains it a little further.

The book A first course in Machine Learning for example does the following steps. First of all it starts from applying Bayes to

$$P(\mathbf{w|t,X)}=\frac{P(\mathbf{t|w,X)}P(\mathbf{w)}}{P(\mathbf{t|X})}$$

where $$\mathbf{w}$$ is the vector of the model parameters, $$\mathbf{X}$$ is the input matrix and $$\mathbf{t}$$ the output vector.

1. My first question is: Why is it starting from this? Should not it start from $$P(\mathbf{t|w,X)}?$$

Then it explains each term in the Bayes expression. Basically it says the numerator can be calculated while the denominator can not. The denominator is re-expressed as

$$P(\mathbf{t|X}) = \int P(\mathbf{t|w,X}) P(\mathbf{w}) d\mathbf{w}$$ and it says

[...] we cannot analytically perform the integration required to compute the marginal likelihood. In other words we have a function $$g(\mathbf{w,X,t}) = P(\mathbf{t|w,X}) P(\mathbf{w})$$ which we know is proportional to the posterior, $$P(\mathbf{w|t,X)}=Zg(\mathbf{w,X,t})$$, but we do not know the constant of proportionality $$Z^{-1}$$.

1. Why it is not possible to perform the integration? Also assuming that when talking about proportionality it is rexpressing the posterior as:$$P(\mathbf{w|t,X)}= \frac{P(\mathbf{t|w,X)}P(\mathbf{w)}}{\int P(\mathbf{t|w,X}) P(\mathbf{w}) d\mathbf{w}}$$, which is not explained, it is not saying which type of proportionality is that (inversely/directly) is it? if so how?

Since we cannot compute these quantities it says we are left with 3 options (as I have understood to find $$\mathbf{w}$$):

1)Find the single value of $$\mathbf{w}$$ that corresponds to the highest value of posterior. As $$g(\mathbf{w,X,t})$$ is proportional to the posterior, a maximum of $$g$$ will correspond to a maximum of the posterior. $$Z^{-1}$$ is not a function of $$w$$.

2)Approximate $$P(\mathbf{w|t,X})$$ with some other density that we can compute analytically.

3) Sample directly from the posterior $$P(\mathbf{w|t,X)}$$ knowing only $$g$$

1. It says the first is not very Bayesian: Why? It gives the example Newton-Raphson method but other books use this method to fit the regression. Also if $$P(\mathbf{w|t,X)}= \frac{P(\mathbf{t|w,X)}P(\mathbf{w)}}{\int P(\mathbf{t|w,X}) P(\mathbf{w}) d\mathbf{w}}$$ is valid, why a maximum in $$g$$ should correspond to a maximum in the posterior? And does logistic regression use a particular method or is it independent from that?

Then it says

Once we have the posterior density, we can predict the response class of new objects by taking an expectation with respect to this density: $$P(t_{new}=1|x_{new}|\mathbf{X,t}) = \mathbf{E}_{P(\mathbf{w|t,X)}}{\frac{1}{1+e^{-\mathbf{w^Tx_{new}}}}}$$

1. Why is the expectation taken over $$P(\mathbf{w|t,X)}$$?

2. What is the first motivation of logistic regression? Does it start from the desire to keep the linearity hence using $$w^Tx$$ but forcing the model to output probabilities so passing the linear function to a sigmoid? Or is this just a consequence of the desire to use log-odds?

1. My first question is: Why is it starting from this? Should not it start from $$P(\mathbf{t|w,X)}?$$

The statement is Bayes theorem. It is used to derive the posterior distribution from prior distribution $$P(\mathbf{w)}$$ and likelihood $$P(\mathbf{t|w,X)}$$. I don't know what do you mean by "starting" from the likelihood, but in Bayesian setting, you need both elements: priors and likelihood.

1. Why it is not possible to perform the integration?

It is not that the integral is "not possible to compute". There is no closed-form mathematical solution for this integral. In plain English, this means that there is no simple mathematical formula for it, where you could just plug-in the data and get the result. That is the reason, why we use methods like MCMC to approximate the integral.

Also assuming that when talking about proportionality it is rexpressing the posterior as: $$P(\mathbf{w|t,X)}= \frac{P(\mathbf{t|w,X)}P(\mathbf{w)}}{\int P(\mathbf{t|w,X}) P(\mathbf{w}) d\mathbf{w}}$$, which is not explained, it is not saying which type of proportionality is that (inversely/directly) is it? if so how?

Hard to say what you mean, but my best guess is that what was meant in the handbook is

$$P(\mathbf{w|t,X)} \propto P(\mathbf{t|w,X)}P(\mathbf{w)}$$

so it is proportional if we drop the normalizing constant $$Z^{-1}$$ (using the notation mentioned by you later).

1. It says the first is not very Bayesian: Why? [...]

Because in Bayesian setting $$\mathbf{w}$$ is a random variable, so it has a distribution and you are interested in finding this distribution, rather then single "best" candidate.

[...] why a maximum in $$g$$ should correspond to a maximum in the posterior?

Because maximum of the posterior distribution is the maximum of the posterior distribution. As said earlier, $$g$$ is proportional to the posterior. In the optimization dividing $$g$$ by $$Z$$ would not make a difference in the result, so they are the same for this purpose. See also Normalizing constant in Bayes theorem, Normalizing constant irrelevant in Bayes theorem?, and Why Normalizing Factor is Required in Bayes Theorem?.

And does logistic regression use a particular method or is it independent from that?

The discussed model is a Bayesian logistic regression model. There is also the "classical" logistic regression that does not use priors and Bayes theorem at all, if this is what you are asking.

1. Why is the expectation taken over $$P(\mathbf{w|t,X)}$$?

Because that is what we are interested in, aren't we? We want to estimate the distribution of the parameters $$\mathbf{w}$$ given the data we observed $$\mathbf{t,X}$$. The procedure is about estimating the parameters of the model.

1. What is the first motivation of logistic regression? Does it start from the desire to keep the linearity hence using $$w^Tx$$ but forcing the model to output probabilities so passing the linear function to a sigmoid?

Yes, that's it. You can think of logistic regression as of linear regression that outputs probabilities, where the target variable is binary (or is a probability of binary event).

• First of all thank you for your response. In 1) I mean that in other references they do not use $P(\mathbf{w|t,X)}$ for the derivation but $P(\mathbf{t|w,X)}$. Then in 2) and 3) what I meant is that it is not obvious (I do not know if incorrect) that $P(\mathbf{w|t,X)} \propto P(\mathbf{t|w,X)}P(\mathbf{w)}$ since at the denominator you have the integral of the numerator. In other words when $P(\mathbf{t|w,X)}P(\mathbf{w)}$increases, so does its integral i.e., the denominator so the direct proportion is not guaranteed in my opinion. – Francesco Boi Sep 23 '19 at 12:42
• Also when I asked if logistic regression use a particular method, I meant does it find a single value for $w$, does it estimate a distribution or does it use sampling ? Or LR is just the model that is independent from the way we find $w$? With the Newton-Raphson method, what kind of logistic regression are we deriving since it is not a very Bayesian method? – Francesco Boi Sep 23 '19 at 12:50
• @FrancescoBoi (1) maybe other references do not discuss Bayesian model? (2) after you compute the integral, $Z$ is a single number (you integrate over all possible $\mathbf{w}$'s and $\mathbf{t,X}$ are constant), so it is dividing by constant; it does not influence optimization. As abput your second comment, maybe this helps: stats.stackexchange.com/questions/163034/… – Tim Sep 23 '19 at 13:01
• Just another question: suppose a closed-form of the integral exist: then the procedure would be to define a loss function, perform its derivative, set to $0$ and verify it is a minimum just as in Linear regression? – Francesco Boi Sep 23 '19 at 13:09
• @FrancescoBoi not exactly. If there was closed form solution, then the posterior distribution would be directly available. Given it, you could easy estimate model of the distribution (maximum), or expected value etc. It would be equivalent to regular logistic regression with flat prior $p(\mathbf{w}) \propto 1$. – Tim Sep 23 '19 at 13:24