I'm learning about a binary classifier. It uses the cross-entropy function as its loss function.
$y_i \log p_i + (1-y_i) \log(1-p_i)$
But why does it use the log function? How about just use linear form as follows?
$y_ip_i + (1-y_i)(1-p_i)$
Is there any advantage to use log function?
And an other question:
Log function maps (0,1) to (-inf, 0). So I think it can crush the algorithm if we get 0 for $p_i$ or $1-p_i$ because log value would be -inf and back-prop will be exploded.
For binary classification one way to encode the probability of an output is $p^y(1-p)^{1-y}$, if y is encoded as 0 or 1. This is the likelihood function and it’s meaning is with probability p we output 0 and with probability 1-p if output is 1.
Now you have a sample and you want to find p which best fits your data. One way is to find the maximum likelihood estimator. If your observations are independent your mle is found by maximizing the likelihood over the whole sample. This is the product of individual likelihoods $\pi_{i=1}^n p^{y_i}(1-p)^{y_i-1}$. But this is hard to use. Because of that one transform likelihood with logs. The transformation is monotonous and you get rid of products and obtain sums which are more tractable. Apply logs and get your expression.
Why not use your encoding instead? I think there is no reason why not. The question is which are the properties of your estimator? The first formulation uses likelihood and mle which has some theory behind which includes the fact that your estimator is efficient. The second formulation is not used often, don’t know any example of encoding the probability like that which does not exclude your approach.
I was also looking for an explanation and found one reason I find intuitive here:
It heavily penalizes predications that are confident and wrong.
Check this graph, it shows the range of possible log loss values given a true observation:
The Log Loss increases rapidly as the predicted probability approaches 0(wrong prediction).
I also asked myself this question some time ago. rapaio explained it already well. I want to add that the log penalizes errors which are farther away from the true label stronger and conversely, errors closer to the true label less. This is not the case with your loss function as it is linear and hence, penalizes equally. You can see that well on Serhiy's image. To your second question, you are right. Therefore it makes sense to add a small epsilon value in order avoid this.
