3
$\begingroup$

I am looking for a proof how to derive the logistic regression from cross-entropy loss, i.e. derive the form of a sigmoid from cross entropy. my thoughts are these:

$\ell = y_i \ln{p_i} + (1-y_i)\ln{(1-p_i)}$

$\frac{\partial \ell}{\partial y_i} = \ln{p_i} - \ln{(1-p_i)} = \ln{\frac{p_i}{1-p_i}}$

and after setting this equal to linear predictor $x'\beta$ I can end up with the sigmoid function. However my question is, what does the derivation of likelihood w.r.t. $y_i$ mean and can I do these operations? In every literature there is only written that we take $\ln{\frac{p_i}{1-p_i}}$ as a link function because it transforms input from $(0,1)$ to real numbers which are desired, but I can not find a connection with cross entropy or Kullback-Leibler Divergence. Thanks.

Improve this question
$\endgroup$
4
  • $\begingroup$ Could you indicate explicitly what you have identified as a "cross entropy?" Also, your question seems circular: since you have started with an expression for the contribution to the log likelihood from each data point, you have already specified the usual form of logistic regression: nothing remains to be derived. $\endgroup$
    – whuber ♦
    Sep 1, 2019 at 16:19
  • $\begingroup$ I would like to start with the likelihood of response $y_i$ : $\ell = p_i^{y_i}(1-p_i)^{(1-y_i)}$, by taking its (negative) log we end up with the cross entropy. And from this I would like to derive the form of a sigmoid function. $\endgroup$
    – pikachu
    Sep 1, 2019 at 16:36
  • 1
    $\begingroup$ You cannot do what you are asking. You have to assume a functional form ( eg ln p/(1-p) is linear function of x.). But there are plenty of other functions... Ehg neural networks assume the function is a sequence of non linear transformation of the input... ( And still use cross entropy with sigmoid on output layer) $\endgroup$
    – seanv507
    Sep 1, 2019 at 21:37
  • $\begingroup$ So the derivative of log-likelihood wrt $y_i$ came in form of log-odds just "by accident" and has nothing to do with what I want? $\endgroup$
    – pikachu
    Sep 2, 2019 at 7:09

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.