Question:
Given a categorical distribution $C_q$ with parameters $q_1, \ldots, q_K$ with $K > 2$, $\sum_k q_k = 1$, which (new) categorical distribution $C_p$ with parameters $p_1, \ldots, p_K$ minimises the weighted cross-entropy loss?
Research:
The Bernoulli case ($K=2$) doesn't seem to be too tricky. There, we suppose that positive weights $w_1$ and $w_2$ are given, and we have one parameter $p$ that we need to find. Suppose $X \sim Bern(q)$ for $0 \leq q \leq 1$ and we have some sampled data $D = \lbrace x_i \rbrace_{i=1...N}$.
Then we have
$$L = \sum_i w_1 (1 - x_i)\log(1 - p) + w_2 x_i \log p$$
$$= w_1\sum_{x_i = 0}\log(1 - p) + w_2\sum_{x_i = 1}\log p$$
$$\approx N(1 - q)w_1\log(1 - p) + Nqw_2\log p.$$
(Could be written more precisely as an expectation, but omitted to reduce notational burden.)
Differentiating w.r.t. $p$ to find the minima we get the following expression: $p_{opt} = \frac{q}{q + \frac{w_1}{w_2}(1 - q)}$.
Generalising this approach to the categorical distribution ($k > 2$) seems a little harder. First partial derivatives w.r.t. each parameter $p_i$ give expressions containing other parameters $p_j$ with $i \neq j$.
Help / thoughts appreciated!
