13
$\begingroup$

I'm learning about a binary classifier. It uses the cross-entropy function as its loss function.

$y_i \log p_i + (1-y_i) \log(1-p_i)$

But why does it use the log function? How about just use linear form as follows?

$y_ip_i + (1-y_i)(1-p_i)$

Is there any advantage to use log function?

And an other question:

Log function maps (0,1) to (-inf, 0). So I think it can crush the algorithm if we get 0 for $p_i$ or $1-p_i$ because log value would be -inf and back-prop will be exploded.

$\endgroup$
12
$\begingroup$

For binary classification one way to encode the probability of an output is $p^y(1-p)^{1-y}$, if y is encoded as 0 or 1. This is the likelihood function and it’s meaning is with probability p we output 0 and with probability 1-p if output is 1.

Now you have a sample and you want to find p which best fits your data. One way is to find the maximum likelihood estimator. If your observations are independent your mle is found by maximizing the likelihood over the whole sample. This is the product of individual likelihoods $\pi_{i=1}^n p^{y_i}(1-p)^{y_i-1}$. But this is hard to use. Because of that one transform likelihood with logs. The transformation is monotonous and you get rid of products and obtain sums which are more tractable. Apply logs and get your expression.

Why not use your encoding instead? I think there is no reason why not. The question is which are the properties of your estimator? The first formulation uses likelihood and mle which has some theory behind which includes the fact that your estimator is efficient. The second formulation is not used often, don’t know any example of encoding the probability like that which does not exclude your approach.

$\endgroup$
2
$\begingroup$

I was also looking for an explanation and found one reason I find intuitive here:

It heavily penalizes predications that are confident and wrong.

Check this graph, it shows the range of possible log loss values given a true observation:

enter image description here

The Log Loss increases rapidly as the predicted probability approaches 0(wrong prediction).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.