# derivative of cross entropy yields log-odds, does that make sense?

I am looking for a proof how to derive the logistic regression from cross-entropy loss, i.e. derive the form of a sigmoid from cross entropy. my thoughts are these:

$$\ell = y_i \ln{p_i} + (1-y_i)\ln{(1-p_i)}$$

$$\frac{\partial \ell}{\partial y_i} = \ln{p_i} - \ln{(1-p_i)} = \ln{\frac{p_i}{1-p_i}}$$

and after setting this equal to linear predictor $$x'\beta$$ I can end up with the sigmoid function. However my question is, what does the derivation of likelihood w.r.t. $$y_i$$ mean and can I do these operations? In every literature there is only written that we take $$\ln{\frac{p_i}{1-p_i}}$$ as a link function because it transforms input from $$(0,1)$$ to real numbers which are desired, but I can not find a connection with cross entropy or Kullback-Leibler Divergence. Thanks.

• Could you indicate explicitly what you have identified as a "cross entropy?" Also, your question seems circular: since you have started with an expression for the contribution to the log likelihood from each data point, you have already specified the usual form of logistic regression: nothing remains to be derived. – whuber Sep 1 '19 at 16:19
• I would like to start with the likelihood of response $y_i$ : $\ell = p_i^{y_i}(1-p_i)^{(1-y_i)}$, by taking its (negative) log we end up with the cross entropy. And from this I would like to derive the form of a sigmoid function. – pikachu Sep 1 '19 at 16:36
• You cannot do what you are asking. You have to assume a functional form ( eg ln p/(1-p) is linear function of x.). But there are plenty of other functions... Ehg neural networks assume the function is a sequence of non linear transformation of the input... ( And still use cross entropy with sigmoid on output layer) – seanv507 Sep 1 '19 at 21:37
• So the derivative of log-likelihood wrt $y_i$ came in form of log-odds just "by accident" and has nothing to do with what I want? – pikachu Sep 2 '19 at 7:09