I am looking for a proof how to derive the logistic regression from cross-entropy loss, i.e. derive the form of a sigmoid from cross entropy. my thoughts are these:

$\ell = y_i \ln{p_i} + (1-y_i)\ln{(1-p_i)}$

$\frac{\partial \ell}{\partial y_i} = \ln{p_i} - \ln{(1-p_i)} = \ln{\frac{p_i}{1-p_i}}$

and after setting this equal to linear predictor $x'\beta$ I can end up with the sigmoid function. However my question is, what does the derivation of likelihood w.r.t. $y_i$ mean and can I do these operations? In every literature there is only written that we take $\ln{\frac{p_i}{1-p_i}}$ as a link function because it transforms input from $(0,1)$ to real numbers which are desired, but I can not find a connection with cross entropy or Kullback-Leibler Divergence. Thanks.

  • $\begingroup$ Could you indicate explicitly what you have identified as a "cross entropy?" Also, your question seems circular: since you have started with an expression for the contribution to the log likelihood from each data point, you have already specified the usual form of logistic regression: nothing remains to be derived. $\endgroup$
    – whuber
    Sep 1, 2019 at 16:19
  • $\begingroup$ I would like to start with the likelihood of response $y_i$ : $\ell = p_i^{y_i}(1-p_i)^{(1-y_i)}$, by taking its (negative) log we end up with the cross entropy. And from this I would like to derive the form of a sigmoid function. $\endgroup$
    – pikachu
    Sep 1, 2019 at 16:36
  • 1
    $\begingroup$ You cannot do what you are asking. You have to assume a functional form ( eg ln p/(1-p) is linear function of x.). But there are plenty of other functions... Ehg neural networks assume the function is a sequence of non linear transformation of the input... ( And still use cross entropy with sigmoid on output layer) $\endgroup$
    – seanv507
    Sep 1, 2019 at 21:37
  • $\begingroup$ So the derivative of log-likelihood wrt $y_i$ came in form of log-odds just "by accident" and has nothing to do with what I want? $\endgroup$
    – pikachu
    Sep 2, 2019 at 7:09


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.