Understanding the Logistic Regression and likelihood

Question

How does the parameter estimation/Training of logistic regression really work? I'll try to put what I've got so far.

1. The output is y the output of the logistic function in form of a probability depending on the value of x : $$P(y=1|x)={1\over1+e^{-\omega^Tx}}\equiv\sigma(\omega^Tx)$$ $$P(y=0|x)=1-P(y=1|x)=1-{1\over1+e^{-\omega^Tx}}$$
2. For one dimension the so called Odds is defined as follows: $${{p(y=1|x)}\over{1-p(y=1|x)}}={{p(y=1|x)}\over{p(y=0|x)}}=e^{\omega_0+\omega_1x}$$
3. Now adding the log function to get the W_0 and W_1 in linear form: $$Logit(y)=log({{p(y=1|x)}\over{1-p(y=1|x)}})=\omega_0+\omega_1x$$
4. Now to the problem part Using the likelihood (Big X is y ) $$L(X|P)=\prod^N_{i=1,y_i=1}P(x_i)\prod^N_{i=1,y_i=0}(1-P(x_i))$$ Can any one tell why we're considering the probability of y=1 twice ? since : $$P(y=0|x)=1-P(y=1|x)$$

and how get the values of ω from it?

Answer 1

Assume in general that you decided to take a model of the form

$$P(y=1|X=x) = h(x;\Theta)$$

for some parameter $\Theta$. Then you simply write down the likelihood for it, i.e.

$$L(\Theta) = \prod_{i \in \{1, ..., N\}, y_i = 1} P(y=1|x=x;\Theta) \cdot \prod_{i \in \{1, ..., N\}, y_i = 0} P(y=0|x=x;\Theta)$$

which is the same as

$$L(\Theta) = \prod_{i \in \{1, ..., N\}, y_i = 1} P(y=1|x=x;\Theta) \cdot \prod_{i \in \{1, ..., N\}, y_i = 0} (1-P(y=1|x=x;\Theta))$$

Now you have decided to 'assume' (model)

$$P(y=1|X=x) = \sigma(\Theta_0 + \Theta_1 x)$$

where $$\sigma(z) = 1/(1+e^{-z})$$

so you just compute the formula for the likelihood and do some kind of optimization algorithm in order to find the $\text{argmax}_\Theta L(\Theta)$, for example, newtons method or any other gradient based method.

Notce that sometimes, people say that when they are doing logistic regression they do not maximize a likelihood (as we/you did above) but rather they minimize a loss function

$$l(\Theta) = -\sum_{i=1}^N{y_i\log(P(Y_i=1|X=x;\Theta)) + (1-y_i)\log(P(Y_i=0|X=x;\Theta))}$$

but notice that $-\log(L(\Theta)) = l(\Theta)$.

This is a general pattern in Machine Learning: The practical side (minimizing loss functions that measure how 'wrong' a heuristic model is) is in fact equal to the 'theoretical side' (modelling explicitly with the $P$-symbol, maximizing statistical quantities like likelihoods) and in fact, many models that do not look like probabilistic ones (SVMs for example) can be reunderstood in a probabilistic context and are in fact maximizations of likelihoods.

Answer 2

Your likelihood function (4) consists of two parts: the product of the probability of success for only those people in your sample who experienced a success, and the product of the probability of failure for only those people in your sample who experienced a failure. Given that each individual experiences either a success or a failure, but not both, the probability will appear for each individual only once. That is what the $, y_i=1$ and $,y_i=0$ mean at the bottom of the product signs.

The coefficients are included in the likelihood function by substituting (1) into (4). That way the likelihood function becomes a function of $\omega$. The point of maximum likelihood is to find the $\omega$ that will maximize the likelihood.