# Which parameters optimise the weighted cross-entropy loss for a pre-specified categorical distribution?

Question:

Given a categorical distribution $$C_q$$ with parameters $$q_1, \ldots, q_K$$ with $$K > 2$$, $$\sum_k q_k = 1$$, which (new) categorical distribution $$C_p$$ with parameters $$p_1, \ldots, p_K$$ minimises the weighted cross-entropy loss?

Research:

The Bernoulli case ($$K=2$$) doesn't seem to be too tricky. There, we suppose that positive weights $$w_1$$ and $$w_2$$ are given, and we have one parameter $$p$$ that we need to find. Suppose $$X \sim Bern(q)$$ for $$0 \leq q \leq 1$$ and we have some sampled data $$D = \lbrace x_i \rbrace_{i=1...N}$$.

Then we have

$$L = \sum_i w_1 (1 - x_i)\log(1 - p) + w_2 x_i \log p$$

$$= w_1\sum_{x_i = 0}\log(1 - p) + w_2\sum_{x_i = 1}\log p$$

$$\approx N(1 - q)w_1\log(1 - p) + Nqw_2\log p.$$

(Could be written more precisely as an expectation, but omitted to reduce notational burden.)

Differentiating w.r.t. $$p$$ to find the minima we get the following expression: $$p_{opt} = \frac{q}{q + \frac{w_1}{w_2}(1 - q)}$$.

Generalising this approach to the categorical distribution ($$k > 2$$) seems a little harder. First partial derivatives w.r.t. each parameter $$p_i$$ give expressions containing other parameters $$p_j$$ with $$i \neq j$$.

Help / thoughts appreciated!

• So you want to minimize the expected weighted cross entropy loss, where the expectation is w.r.t. some $X$ which takes $K$ values. Is this correct? Sep 7 at 17:56
• @picky_porpoise, correct! and where we also want to know which $Y$ (also taking $K$ values) is the minimiser. Sep 8 at 6:36
• Then my intuition would be to use a change of measure. Transform the distribution of $X$ via the weights and then the optimization w.r.t. the new measure should be easier. Sep 9 at 12:20