# Simplification of case-based logistic regression cost function

In Andrew Ng's Coursera Machine Learning class, the cost function for logistic regression is defined as:

\begin{align*} J(\theta) &= \dfrac{1}{m} \sum_{i=1}^m \mathrm{Cost}(h_\theta(x^{(i)}),y^{(i)}) \\[2ex] \mathrm{Cost}(h_\theta(x),y) &= \begin{cases} -\log(1-h_\theta(x)) \; & \text{if y = 0} \\ -\log(h_\theta(x)) \; & \text{if y = 1} \\ \end{cases} \end{align*}

It is said that this simplifies to:

$$\require{cancel} J(\theta) = - \frac{1}{m} \displaystyle \sum_{i=1}^m \left[y^{(i)}\log \big(h_\theta (x^{(i)})\big) + (1 - y^{(i)})\log \big(1 - h_\theta(x^{(i)})\big)\right] \\[6pt]$$

How is this simpler? Why are the cases multiplied by $y^{(i)}$ and $1-y^{(i)}$ respectively?

• Re: "Why are the cases multiplied by y(i) and 1−y(i) respectively?" -- what does the expression inside the sum work out to when $y(i)=1$? What about when $y(i)=0$? ... it's not hard to work out. – Glen_b Jun 29 '17 at 13:03

The single case formula is simpler in the sense that it can be written on one line.

The implementation may still be more efficient using branching based on the value of $y^{(i)}$ to avoid unnecessary calculation.

$y^{(i)} \in \{0,1\}$, so $y^{(i)}$ and $1-y^{(i)}$ will evaluate to either $0$ or $1$.

Multiplying a case by $0$ cancels it, and multiplying by $1$ keeps it.

Given: $$\require{cancel} J(\theta) = - \frac{1}{m} \displaystyle \sum_{i=1}^m \left[y^{(i)}\log \big(h_\theta (x^{(i)})\big) + (1 - y^{(i)})\log \big(1 - h_\theta(x^{(i)})\big)\right] \\[6pt]$$

$$J(\theta) = \begin{cases} \displaystyle - \frac{1}{m} \sum_{i=1}^m \left[\cancel{0 \cdot \log (h_\theta (x^{(i)}))} + \cancel{(1 - 0)}\log (1 - h_\theta(x^{(i)}))\right] \quad \text{if y=0}\\ \displaystyle - \frac{1}{m} \sum_{i=1}^m \left[y^{(i)}\log (h_\theta (x^{(i)})) + \cancel{(1-1) \log (1 - h_\theta(x^{(i)}))}\right] \quad \text{if y=1}\\ \end{cases}$$

Which is the same as the original case-defined definition (moving the negation before the $1 \over m$).