I am looking for a proof how to derive the logistic regression from cross-entropy loss, i.e. derive the form of a sigmoid from cross entropy. my thoughts are these:

$\ell = y_i \ln{p_i} + (1-y_i)\ln{(1-p_i)}$

$\frac{\partial \ell}{\partial y_i} = \ln{p_i} - \ln{(1-p_i)} = \ln{\frac{p_i}{1-p_i}}$

and after setting this equal to linear predictor $x'\beta$ I can end up with the sigmoid function. However my question is, what does the derivation of likelihood w.r.t. $y_i$ mean and can I do these operations? In every literature there is only written that we take $\ln{\frac{p_i}{1-p_i}}$ as a link function because it transforms input from $(0,1)$ to real numbers which are desired, but I can not find a connection with cross entropy or Kullback-Leibler Divergence. Thanks.

  • $\begingroup$ Could you indicate explicitly what you have identified as a "cross entropy?" Also, your question seems circular: since you have started with an expression for the contribution to the log likelihood from each data point, you have already specified the usual form of logistic regression: nothing remains to be derived. $\endgroup$ – whuber Sep 1 '19 at 16:19
  • $\begingroup$ I would like to start with the likelihood of response $y_i$ : $\ell = p_i^{y_i}(1-p_i)^{(1-y_i)}$, by taking its (negative) log we end up with the cross entropy. And from this I would like to derive the form of a sigmoid function. $\endgroup$ – pikachu Sep 1 '19 at 16:36
  • 1
    $\begingroup$ You cannot do what you are asking. You have to assume a functional form ( eg ln p/(1-p) is linear function of x.). But there are plenty of other functions... Ehg neural networks assume the function is a sequence of non linear transformation of the input... ( And still use cross entropy with sigmoid on output layer) $\endgroup$ – seanv507 Sep 1 '19 at 21:37
  • $\begingroup$ So the derivative of log-likelihood wrt $y_i$ came in form of log-odds just "by accident" and has nothing to do with what I want? $\endgroup$ – pikachu Sep 2 '19 at 7:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.