0
$\begingroup$

In Andrew Ng's Coursera Machine Learning class, the cost function for logistic regression is defined as:

$$\begin{align*} J(\theta) &= \dfrac{1}{m} \sum_{i=1}^m \mathrm{Cost}(h_\theta(x^{(i)}),y^{(i)}) \\[2ex] \mathrm{Cost}(h_\theta(x),y) &= \begin{cases} -\log(1-h_\theta(x)) \; & \text{if y = 0} \\ -\log(h_\theta(x)) \; & \text{if y = 1} \\ \end{cases} \end{align*}$$

It is said that this simplifies to:

$$\require{cancel} J(\theta) = - \frac{1}{m} \displaystyle \sum_{i=1}^m \left[y^{(i)}\log \big(h_\theta (x^{(i)})\big) + (1 - y^{(i)})\log \big(1 - h_\theta(x^{(i)})\big)\right] \\[6pt]$$

How is this simpler? Why are the cases multiplied by $y^{(i)}$ and $1-y^{(i)}$ respectively?

$\endgroup$
  • $\begingroup$ Re: "Why are the cases multiplied by y(i) and 1−y(i) respectively?" -- what does the expression inside the sum work out to when $y(i)=1$? What about when $y(i)=0$? ... it's not hard to work out. $\endgroup$ – Glen_b -Reinstate Monica Jun 29 '17 at 13:03
0
$\begingroup$

The single case formula is simpler in the sense that it can be written on one line.

The implementation may still be more efficient using branching based on the value of $y^{(i)}$ to avoid unnecessary calculation.

$y^{(i)} \in \{0,1\}$, so $y^{(i)}$ and $1-y^{(i)}$ will evaluate to either $0$ or $1$.

Multiplying a case by $0$ cancels it, and multiplying by $1$ keeps it.

Given: $$\require{cancel} J(\theta) = - \frac{1}{m} \displaystyle \sum_{i=1}^m \left[y^{(i)}\log \big(h_\theta (x^{(i)})\big) + (1 - y^{(i)})\log \big(1 - h_\theta(x^{(i)})\big)\right] \\[6pt]$$

$$J(\theta) = \begin{cases} \displaystyle - \frac{1}{m} \sum_{i=1}^m \left[\cancel{0 \cdot \log (h_\theta (x^{(i)}))} + \cancel{(1 - 0)}\log (1 - h_\theta(x^{(i)}))\right] \quad \text{if $y=0$}\\ \displaystyle - \frac{1}{m} \sum_{i=1}^m \left[y^{(i)}\log (h_\theta (x^{(i)})) + \cancel{(1-1) \log (1 - h_\theta(x^{(i)}))}\right] \quad \text{if $y=1$}\\ \end{cases}$$

Which is the same as the original case-defined definition (moving the negation before the $1 \over m$).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.