38
$\begingroup$

I am doing the Machine Learning Stanford course on Coursera.

In the chapter on Logistic Regression, the cost function is this: enter image description here

Then, it is differentiated here: enter image description here

I tried getting the derivative of the cost function, but I got something completely different.

How is the derivative obtained?

Which are the intermediary steps?

$\endgroup$
5
  • $\begingroup$ +1, check @AdamO's answer in my question here. stats.stackexchange.com/questions/229014/… $\endgroup$
    – Haitao Du
    May 10 '17 at 18:39
  • $\begingroup$ "Completely different" is not really sufficient to answer your question, besides telling you what you already know (the correct gradient). It'd be much more useful if you gave us what your calculations resulted in, then we can help you shore up where you made the mistake. $\endgroup$ May 10 '17 at 20:43
  • $\begingroup$ @MatthewDrury Sorry, Matt, I had arranged the answer right before your comment came in. Octavian, did you follow all the steps? I will edit to give it some added value later... $\endgroup$ May 10 '17 at 20:46
  • 2
    $\begingroup$ when you say "derivated" do you mean "differentiated" or "derived"? $\endgroup$
    – Glen_b
    May 11 '17 at 3:19
  • $\begingroup$ Here is another, in my opinion easy to follow, explanation of how the partial derivatives of the logistic regression cost function can be obtained. $\endgroup$
    – guestguest
    Nov 26 '20 at 19:31
52
$\begingroup$

Adapted from the notes in the course, which I don't see available (including this derivation) outside the notes contributed by students within the page of Andrew Ng's Coursera Machine Learning course.


In what follows, the superscript $(i)$ denotes individual measurements or training "examples."

$\small \frac{\partial J(\theta)}{\partial \theta_j} = \frac{\partial}{\partial \theta_j} \,\frac{-1}{m}\sum_{i=1}^m \left[ y^{(i)}\log\left(h_\theta \left(x^{(i)}\right)\right) + (1 -y^{(i)})\log\left(1-h_\theta \left(x^{(i)}\right)\right)\right] \\[2ex]\small\underset{\text{linearity}}= \,\frac{-1}{m}\,\sum_{i=1}^m \left[ y^{(i)}\frac{\partial}{\partial \theta_j}\log\left(h_\theta \left(x^{(i)}\right)\right) + (1 -y^{(i)})\frac{\partial}{\partial \theta_j}\log\left(1-h_\theta \left(x^{(i)}\right)\right) \right] \\[2ex]\Tiny\underset{\text{chain rule}}= \,\frac{-1}{m}\,\sum_{i=1}^m \left[ y^{(i)}\frac{\frac{\partial}{\partial \theta_j}h_\theta \left(x^{(i)}\right)}{h_\theta\left(x^{(i)}\right)} + (1 -y^{(i)})\frac{\frac{\partial}{\partial \theta_j}\left(1-h_\theta \left(x^{(i)}\right)\right)}{1-h_\theta\left(x^{(i)}\right)} \right] \\[2ex]\small\underset{h_\theta(x)=\sigma\left(\theta^\top x\right)}=\,\frac{-1}{m}\,\sum_{i=1}^m \left[ y^{(i)}\frac{\frac{\partial}{\partial \theta_j}\sigma\left(\theta^\top x^{(i)}\right)}{h_\theta\left(x^{(i)}\right)} + (1 -y^{(i)})\frac{\frac{\partial}{\partial \theta_j}\left(1-\sigma\left(\theta^\top x^{(i)}\right)\right)}{1-h_\theta\left(x^{(i)}\right)} \right] \\[2ex]\Tiny\underset{\sigma'}=\frac{-1}{m}\,\sum_{i=1}^m \left[ y^{(i)}\, \frac{\sigma\left(\theta^\top x^{(i)}\right)\left(1-\sigma\left(\theta^\top x^{(i)}\right)\right)\frac{\partial}{\partial \theta_j}\left(\theta^\top x^{(i)}\right)}{h_\theta\left(x^{(i)}\right)} - (1 -y^{(i)})\,\frac{\sigma\left(\theta^\top x^{(i)}\right)\left(1-\sigma\left(\theta^\top x^{(i)}\right)\right)\frac{\partial}{\partial \theta_j}\left(\theta^\top x^{(i)}\right)}{1-h_\theta\left(x^{(i)}\right)} \right] \\[2ex]\small\underset{\sigma\left(\theta^\top x\right)=h_\theta(x)}= \,\frac{-1}{m}\,\sum_{i=1}^m \left[ y^{(i)}\frac{h_\theta\left( x^{(i)}\right)\left(1-h_\theta\left( x^{(i)}\right)\right)\frac{\partial}{\partial \theta_j}\left(\theta^\top x^{(i)}\right)}{h_\theta\left(x^{(i)}\right)} - (1 -y^{(i)})\frac{h_\theta\left( x^{(i)}\right)\left(1-h_\theta\left(x^{(i)}\right)\right)\frac{\partial}{\partial \theta_j}\left( \theta^\top x^{(i)}\right)}{1-h_\theta\left(x^{(i)}\right)} \right] \\[2ex]\small\underset{\frac{\partial}{\partial \theta_j}\left(\theta^\top x^{(i)}\right)=x_j^{(i)}}=\,\frac{-1}{m}\,\sum_{i=1}^m \left[y^{(i)}\left(1-h_\theta\left(x^{(i)}\right)\right)x_j^{(i)}- \left(1-y^{i}\right)\,h_\theta\left(x^{(i)}\right)x_j^{(i)} \right] \\[2ex]\small\underset{\text{distribute}}=\,\frac{-1}{m}\,\sum_{i=1}^m \left[y^{i}-y^{i}h_\theta\left(x^{(i)}\right)- h_\theta\left(x^{(i)}\right)+y^{(i)}h_\theta\left(x^{(i)}\right) \right]\,x_j^{(i)} \\[2ex]\small\underset{\text{cancel}}=\,\frac{-1}{m}\,\sum_{i=1}^m \left[y^{(i)}-h_\theta\left(x^{(i)}\right)\right]\,x_j^{(i)} \\[2ex]\small=\frac{1}{m}\sum_{i=1}^m\left[h_\theta\left(x^{(i)}\right)-y^{(i)}\right]\,x_j^{(i)} $


The derivative of the sigmoid function is

$\Tiny\begin{align}\frac{d}{dx}\sigma(x)&=\frac{d}{dx}\left(\frac{1}{1+e^{-x}}\right)\\[2ex] &=\frac{-(1+e^{-x})'}{(1+e^{-x})^2}\\[2ex] &=\frac{e^{-x}}{(1+e^{-x})^2}\\[2ex] &=\left(\frac{1}{1+e^{-x}}\right)\left(\frac{e^{-x}}{1+e^{-x}}\right)\\[2ex] &=\left(\frac{1}{1+e^{-x}}\right)\,\left(\frac{1+e^{-x}}{1+e^{-x}}-\frac{1}{1+e^{-x}}\right)\\[2ex] &=\sigma(x)\,\left(\frac{1+e^{-x}}{1+e^{-x}}-\sigma(x)\right)\\[2ex] &=\sigma(x)\,(1-\sigma(x)) \end{align}$

$\endgroup$
17
  • 1
    $\begingroup$ +1 for all the efforts!, may be using matrix notation could be easier? $\endgroup$
    – Haitao Du
    May 11 '17 at 1:43
  • 1
    $\begingroup$ @hxd1011 Ty! Yes, no more $\sum$'s, but wanted to be consistent with OP. $\endgroup$ May 11 '17 at 1:46
  • 2
    $\begingroup$ that is why I appreciate your effort. you spend time to us OP's language!! $\endgroup$
    – Haitao Du
    May 11 '17 at 1:47
  • 2
    $\begingroup$ My understanding is that there are convexity issues that make the squared error minimization undesirable for non-linear activation functions. In matrix notation, it would be $\frac{\partial J(\theta)}{\partial \theta}=\frac{1}{m}X^\top\left( \sigma(X\theta)-\mathbf y\right)$. $\endgroup$ May 11 '17 at 1:57
  • 1
    $\begingroup$ @MohammedNoureldin I just took the partial derivative in the numerators on the prior line, applying the chain rule. $\endgroup$ Feb 25 '19 at 15:32
13
$\begingroup$

To avoid impression of excessive complexity of the matter, let us just see the structure of solution.

With simplification and some abuse of notation, let $G(\theta)$ be a term in sum of $J(\theta)$, and $h = 1/(1+e^{-z})$ is a function of $z(\theta)= x \theta $: $$ G = y \cdot \log(h)+(1-y)\cdot \log(1-h) $$

We may use chain rule: $\frac{d G}{d \theta}=\frac{d G}{d h}\frac{d h}{d z}\frac{d z}{d \theta}$ and solve it one by one ($x$ and $y$ are constants).

$$\frac{d G}{\partial h} = \frac{y} {h} - \frac{1-y}{1-h} = \frac{y - h}{h(1-h)} $$ For sigmoid $\frac{d h}{d z} = h (1-h) $ holds, which is just a denominator of the previous statement.

Finally, $\frac{d z}{d \theta} = x $.

Combining results all together gives sought-for expression: $$\frac{d G}{d \theta} = (y-h)x $$ Hope that helps.

$\endgroup$
1
  • 1
    $\begingroup$ This is just fine. I was having a hard time converting this into a matrix notation. $\endgroup$ Jul 18 at 18:22
1
$\begingroup$

For those of us who are not so strong at calculus, but would like to play around with adjusting the cost function and need to find a way to calculate derivatives... a short cut to re-learning calculus is this online tool to automatically provide the derivation, with step by step explanations of the rule.

https://www.derivative-calculator.net

Example of deriving cost function of sigmoid activation in logistic regression

$\endgroup$
0
$\begingroup$

The credit for this answer goes to Antoni Parellada from the comments, which I think deserves a more prominent place on this page (as it helped me out when many other answers did not). Also, this is not a full derivation but more of a clear statement of $\frac{\partial J(\theta)}{\partial \theta}$. (For full derivation, see the other answers).

$$\frac{\partial J(\theta)}{\partial \theta} = \frac{1}{m} \cdot X^T\big(\sigma(X\theta)-y\big)$$

where

\begin{equation} \begin{aligned} X \in \mathbb{R}^{m\times n} &= \text{Training example matrix} \\ \sigma(z) &= \frac{1}{1+e^{-z}} = \text{sigmoid function} = \text{logistic function} \\ \theta \in \mathbb{R}^{n} &= \text{weight row vector} \\ y &= \text{class/category/label corresponding to rows in X} \end{aligned} \end{equation}

Also, a Python implementation for those wanting to calculate the gradient of $J$ with respect to $\theta$.

import numpy
def sig(z):
return 1/(1+np.e**-(z))


def compute_grad(X, y, w):
    """
    Compute gradient of cross entropy function with sigmoidal probabilities

    Args: 
        X (numpy.ndarray): examples. Individuals in rows, features in columns
        y (numpy.ndarray): labels. Vector corresponding to rows in X
        w (numpy.ndarray): weight vector

    Returns: 
        numpy.ndarray 

    """
    m = X.shape[0]
    Z = w.dot(X.T)
    A = sig(Z)
    return  (-1/ m) * (X.T * (A - y)).sum(axis=1) 
$\endgroup$
0
$\begingroup$

Another presentation, with matrix notation.

Preparation: $\sigma(t)=\frac{1}{1+e^{-t}}$ has $\frac{d \ln \sigma(t)}{dt}=\sigma(-t)=1-\sigma(t)$ hence $\frac{d \sigma}{dt}=\sigma(1-\sigma)$ and hence $\frac{d \ln (1- \sigma)}{dt}=\sigma$.

We use the convention in which all vectors are column vectors. Let $X$ be the data matrix whose rows are the data points $x_i^T$. Using the convention that a scalar function applying to a vector is applied entry-wise, we have

$$mJ(\theta)=\sum_i -y_i \ln \sigma(x_i^T\theta)-(1-y_i) \ln (1-\sigma(x_i^T\theta))=-y^T \ln \sigma (X\theta)-(1^T-y^T)\ln(1-\sigma)(X\theta).$$

Now the derivative (Jacobian, row vector) of $J$ with respect to $ \theta$ is obtained by using chain rule and noting that for matrix $M$, column vector $v$ and $f$ acting entry-wise we have $D_v f(Mv)=\text{diag}(f'(Mv))M$. The computation is as follows:

$$m D_\theta J= -y^T [\text{diag}((1-\sigma)(X\theta))] X-(1^T-y^T) [\text{diag}(-\sigma(X\theta))]X=$$ $$=-y^TX+1^T[\text{diag}(\sigma(X\theta))]X=-y^TX+(\sigma(X\theta))^TX.$$

Finally, the gradient is

$$\nabla_\theta J=(D_\theta J)^T=\frac{1}{m}X^T(\sigma(X\theta)-y)$$

$\endgroup$
1
  • $\begingroup$ This is an example of a generalized linear model with canonical activation function See also Bishop, "Pattern Recognition and Machine Learning", Section 4.3.6, p.212. $\endgroup$
    – Max M
    Jul 8 at 0:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.