Simplification of case-based logistic regression cost function

Question

In Andrew Ng's Coursera Machine Learning class, the cost function for logistic regression is defined as:

$$\begin{align*} J(\theta) &= \dfrac{1}{m} \sum_{i=1}^m \mathrm{Cost}(h_\theta(x^{(i)}),y^{(i)}) \\[2ex] \mathrm{Cost}(h_\theta(x),y) &= \begin{cases} -\log(1-h_\theta(x)) \; & \text{if y = 0} \\ -\log(h_\theta(x)) \; & \text{if y = 1} \\ \end{cases} \end{align*}$$

It is said that this simplifies to:

$$\require{cancel} J(\theta) = - \frac{1}{m} \displaystyle \sum_{i=1}^m \left[y^{(i)}\log \big(h_\theta (x^{(i)})\big) + (1 - y^{(i)})\log \big(1 - h_\theta(x^{(i)})\big)\right] \\[6pt]$$

How is this simpler? Why are the cases multiplied by $y^{(i)}$ and $1-y^{(i)}$ respectively?

Answer 1

The single case formula is simpler in the sense that it can be written on one line.

The implementation may still be more efficient using branching based on the value of $y^{(i)}$ to avoid unnecessary calculation.

$y^{(i)} \in \{0,1\}$, so $y^{(i)}$ and $1-y^{(i)}$ will evaluate to either $0$ or $1$.

Multiplying a case by $0$ cancels it, and multiplying by $1$ keeps it.

Given: $$\require{cancel} J(\theta) = - \frac{1}{m} \displaystyle \sum_{i=1}^m \left[y^{(i)}\log \big(h_\theta (x^{(i)})\big) + (1 - y^{(i)})\log \big(1 - h_\theta(x^{(i)})\big)\right] \\[6pt]$$

$$J(\theta) = \begin{cases} \displaystyle - \frac{1}{m} \sum_{i=1}^m \left[\cancel{0 \cdot \log (h_\theta (x^{(i)}))} + \cancel{(1 - 0)}\log (1 - h_\theta(x^{(i)}))\right] \quad \text{if $y=0$}\\ \displaystyle - \frac{1}{m} \sum_{i=1}^m \left[y^{(i)}\log (h_\theta (x^{(i)})) + \cancel{(1-1) \log (1 - h_\theta(x^{(i)}))}\right] \quad \text{if $y=1$}\\ \end{cases}$$

Which is the same as the original case-defined definition (moving the negation before the $1 \over m$).