Given a categorical distribution $C_q$ with parameters $q_1, \ldots, q_K$ with $K > 2$, $\sum_k q_k = 1$, which (new) categorical distribution $C_p$ with parameters $p_1, \ldots, p_K$ minimises the weighted cross-entropy loss?


The Bernoulli case ($K=2$) doesn't seem to be too tricky. There, we suppose that positive weights $w_1$ and $w_2$ are given, and we have one parameter $p$ that we need to find. Suppose $X \sim Bern(q)$ for $0 \leq q \leq 1$ and we have some sampled data $D = \lbrace x_i \rbrace_{i=1...N}$.

Then we have

$$L = \sum_i w_1 (1 - x_i)\log(1 - p) + w_2 x_i \log p$$

$$= w_1\sum_{x_i = 0}\log(1 - p) + w_2\sum_{x_i = 1}\log p$$

$$\approx N(1 - q)w_1\log(1 - p) + Nqw_2\log p.$$

(Could be written more precisely as an expectation, but omitted to reduce notational burden.)

Differentiating w.r.t. $p$ to find the minima we get the following expression: $p_{opt} = \frac{q}{q + \frac{w_1}{w_2}(1 - q)}$.

Generalising this approach to the categorical distribution ($k > 2$) seems a little harder. First partial derivatives w.r.t. each parameter $p_i$ give expressions containing other parameters $p_j$ with $i \neq j$.

Help / thoughts appreciated!

  • $\begingroup$ So you want to minimize the expected weighted cross entropy loss, where the expectation is w.r.t. some $X$ which takes $K$ values. Is this correct? $\endgroup$ Sep 7 at 17:56
  • $\begingroup$ @picky_porpoise, correct! and where we also want to know which $Y$ (also taking $K$ values) is the minimiser. $\endgroup$
    – montol
    Sep 8 at 6:36
  • $\begingroup$ Then my intuition would be to use a change of measure. Transform the distribution of $X$ via the weights and then the optimization w.r.t. the new measure should be easier. $\endgroup$ Sep 9 at 12:20


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.