How does the parameter estimation/Training of logistic regression really work? I'll try to put what I've got so far.

  1. The output is y the output of the logistic function in form of a probability depending on the value of x : $$P(y=1|x)={1\over1+e^{-\omega^Tx}}\equiv\sigma(\omega^Tx)$$ $$P(y=0|x)=1-P(y=1|x)=1-{1\over1+e^{-\omega^Tx}}$$
  2. For one dimension the so called Odds is defined as follows: $${{p(y=1|x)}\over{1-p(y=1|x)}}={{p(y=1|x)}\over{p(y=0|x)}}=e^{\omega_0+\omega_1x}$$
  3. Now adding the log function to get the W_0 and W_1 in linear form: $$Logit(y)=log({{p(y=1|x)}\over{1-p(y=1|x)}})=\omega_0+\omega_1x$$
  4. Now to the problem part Using the likelihood (Big X is y ) $$L(X|P)=\prod^N_{i=1,y_i=1}P(x_i)\prod^N_{i=1,y_i=0}(1-P(x_i))$$ Can any one tell why we're considering the probability of y=1 twice ? since : $$P(y=0|x)=1-P(y=1|x)$$

and how get the values of ω from it?


2 Answers 2


Assume in general that you decided to take a model of the form

$$P(y=1|X=x) = h(x;\Theta)$$

for some parameter $\Theta$. Then you simply write down the likelihood for it, i.e.

$$L(\Theta) = \prod_{i \in \{1, ..., N\}, y_i = 1} P(y=1|x=x;\Theta) \cdot \prod_{i \in \{1, ..., N\}, y_i = 0} P(y=0|x=x;\Theta)$$

which is the same as

$$L(\Theta) = \prod_{i \in \{1, ..., N\}, y_i = 1} P(y=1|x=x;\Theta) \cdot \prod_{i \in \{1, ..., N\}, y_i = 0} (1-P(y=1|x=x;\Theta))$$

Now you have decided to 'assume' (model)

$$P(y=1|X=x) = \sigma(\Theta_0 + \Theta_1 x)$$

where $$\sigma(z) = 1/(1+e^{-z})$$

so you just compute the formula for the likelihood and do some kind of optimization algorithm in order to find the $\text{argmax}_\Theta L(\Theta)$, for example, newtons method or any other gradient based method.

Notce that sometimes, people say that when they are doing logistic regression they do not maximize a likelihood (as we/you did above) but rather they minimize a loss function

$$l(\Theta) = -\sum_{i=1}^N{y_i\log(P(Y_i=1|X=x;\Theta)) + (1-y_i)\log(P(Y_i=0|X=x;\Theta))}$$

but notice that $-\log(L(\Theta)) = l(\Theta)$.

This is a general pattern in Machine Learning: The practical side (minimizing loss functions that measure how 'wrong' a heuristic model is) is in fact equal to the 'theoretical side' (modelling explicitly with the $P$-symbol, maximizing statistical quantities like likelihoods) and in fact, many models that do not look like probabilistic ones (SVMs for example) can be reunderstood in a probabilistic context and are in fact maximizations of likelihoods.

  • $\begingroup$ @Werner thanks for your answer. But I still need a bit of clarification.1st can you please explain what on earth the 2 $\prod$ stay for in the definition of $L(\theta)$ since as far I understood it I'm interessted in the case of $y_i =1 $. and how can get the values of $\omega_1$ and $\omega_0$ thanks a lot dor your help ! $\endgroup$
    – Engine
    Sep 26, 2017 at 16:16
  • $\begingroup$ @Engine: The big 'pi' is a product... like a big Sigma $\Sigma$ is a sum... do you understand or do you need more clarification on that as well? On the second question: Lets say we want to minimize a function $f(x) = x^2$ and we start at $x=3$ but let us assume that we do not know/can not express / can not visualize $f$ as it is to complicated. Now the derivative of $f$ is $f' = 2x$. Interestingly if we are right from the minimum $x=0$ it points to the right and if we are left of it it points left. Mathematically the derivative points into the direction of the 'strongest ascend' $\endgroup$ Sep 27, 2017 at 6:10
  • $\begingroup$ @Engine: In more dimensions you replace the derivative by the gradient, i.e. you start off at a random point $x_0$ and compute the gradient $\partial f$ at $x$ and if you want to maximize then your next point $x_1$ is $x_1 = x_0 + \partial f(x_0)$. Then you compute $\partial f(x_1)$ and you next $x$ is $x_2 = x_1 + \partial f(x_1)$ and so forth. This is called gradient ascend/descent and is the most common technique in maximizing a function. Now you do that with $L(\Theta)$ or in your notation $L(\omega)$ in order to find the $\omega$ that maxeimizes $L$ $\endgroup$ Sep 27, 2017 at 6:13
  • $\begingroup$ @Engine: You are not at all interested in the case $y=1$! You are interested in 'the' $\omega$ that 'best explains your data'. From thet $\omega$ aou let the model 'speak for itself' and get back to the case of $y=1$ but first of all you need to setup a model! Here, 'best explains' means 'having the highest likelihood' because that is what people came up with (and I think it is very natural)... however, there are other metrics (different loss functions and so on) that one could use! There are two products because we want the model to explain the $y=1$ as well as the $y=0$ 'good'! $\endgroup$ Sep 27, 2017 at 6:16

Your likelihood function (4) consists of two parts: the product of the probability of success for only those people in your sample who experienced a success, and the product of the probability of failure for only those people in your sample who experienced a failure. Given that each individual experiences either a success or a failure, but not both, the probability will appear for each individual only once. That is what the $, y_i=1$ and $,y_i=0$ mean at the bottom of the product signs.

The coefficients are included in the likelihood function by substituting (1) into (4). That way the likelihood function becomes a function of $\omega$. The point of maximum likelihood is to find the $\omega$ that will maximize the likelihood.

  • $\begingroup$ thanks so much for your answer, sorry but still don't get it. isn't$y_i = 0 $ means the probability that y =0[Don't occure] for all y's of the product. and vis versa for y_i=1. And still after the subtitutiing of how can I find $\omega$ values, caclulating the 2nd derivative ? or gradient ? thanks a lot for your help ! $\endgroup$
    – Engine
    Sep 26, 2017 at 15:41
  • $\begingroup$ $\prod_{i=1, y=1}^N$ should be read as "product for persons $i=1$ till $N$, but only if $y=1$. So the first part only applies to those persons in your data that experienced the event. Similarly, the second part only refers to persons who did not experienced the event. $\endgroup$ Sep 26, 2017 at 18:16
  • $\begingroup$ There are many possible algorithms for maximizing the likelihood function. The most common one, the Newton-Raphson method, indeed involves computing the first and second derivatives. $\endgroup$ Sep 26, 2017 at 18:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.