In Andrew Ng's Coursera Machine Learning class, the cost function for logistic regression is defined as:

$$\begin{align*} J(\theta) &= \dfrac{1}{m} \sum_{i=1}^m \mathrm{Cost}(h_\theta(x^{(i)}),y^{(i)}) \\[2ex] \mathrm{Cost}(h_\theta(x),y) &= \begin{cases} -\log(1-h_\theta(x)) \; & \text{if y = 0} \\ -\log(h_\theta(x)) \; & \text{if y = 1} \\ \end{cases} \end{align*}$$

It is said that this simplifies to:

$$\require{cancel} J(\theta) = - \frac{1}{m} \displaystyle \sum_{i=1}^m \left[y^{(i)}\log \big(h_\theta (x^{(i)})\big) + (1 - y^{(i)})\log \big(1 - h_\theta(x^{(i)})\big)\right] \\[6pt]$$

How is this simpler? Why are the cases multiplied by $y^{(i)}$ and $1-y^{(i)}$ respectively?

  • $\begingroup$ Re: "Why are the cases multiplied by y(i) and 1−y(i) respectively?" -- what does the expression inside the sum work out to when $y(i)=1$? What about when $y(i)=0$? ... it's not hard to work out. $\endgroup$
    – Glen_b
    Jun 29, 2017 at 13:03

1 Answer 1


The single case formula is simpler in the sense that it can be written on one line.

The implementation may still be more efficient using branching based on the value of $y^{(i)}$ to avoid unnecessary calculation.

$y^{(i)} \in \{0,1\}$, so $y^{(i)}$ and $1-y^{(i)}$ will evaluate to either $0$ or $1$.

Multiplying a case by $0$ cancels it, and multiplying by $1$ keeps it.

Given: $$\require{cancel} J(\theta) = - \frac{1}{m} \displaystyle \sum_{i=1}^m \left[y^{(i)}\log \big(h_\theta (x^{(i)})\big) + (1 - y^{(i)})\log \big(1 - h_\theta(x^{(i)})\big)\right] \\[6pt]$$

$$J(\theta) = \begin{cases} \displaystyle - \frac{1}{m} \sum_{i=1}^m \left[\cancel{0 \cdot \log (h_\theta (x^{(i)}))} + \cancel{(1 - 0)}\log (1 - h_\theta(x^{(i)}))\right] \quad \text{if $y=0$}\\ \displaystyle - \frac{1}{m} \sum_{i=1}^m \left[y^{(i)}\log (h_\theta (x^{(i)})) + \cancel{(1-1) \log (1 - h_\theta(x^{(i)}))}\right] \quad \text{if $y=1$}\\ \end{cases}$$

Which is the same as the original case-defined definition (moving the negation before the $1 \over m$).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.