1
$\begingroup$

Let $\{y_1...y_n\} \in \{0,1\}$, and let $c \in [0,1]$. Define the cross-entropy of loss of $c$ by:

$$C(c): = \sum_{j=1}^{n}- y_j ln c - (1- y_j) ln (1-c) $$.

Define $c*= arg min _{c} C(c)$

Is there an analytical expression of this $c*$, in terms of $\{y_1...y_n\}$? I'm asking this question motivated by other two loss functions: $L^1, L^2$ loss, i.e. quadratic and absolute losses for $c$, where the median and the mean of $\{y_1...y_n\}$ are the corresponding minimizers of $C(c)$.

$\endgroup$

1 Answer 1

1
$\begingroup$

You're effectively interested in the Maximum Likelihood Estimate of a Bernoulli random variable with parameter $p=c$.

Taking the derivative in $c$ gives $-\sum_{j=1}^n y_j/c +(1-y_j)/(1-c) = 0.$

Define $Y_n:= \sum_{j=1}^n y_j$. Then

$$-\frac{1}{c}Y_n+(n-Y_n)\frac{1}{1-c}=0,$$

giving:

$$c=\frac{Y_n}{n}.$$

Taking a step back, this should be obvious, as the numerator counts the number of successes $(y_i=1)$ and the denominator counts the number of trials.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.