2
$\begingroup$

What is the best way to control the entropy of a categorical probability distribution?

I have a categorical probability distribution $p_1, p_2, \dots, p_K$, for $K$ small-ish ($< 1000$). Assume that the probabilities are generally distinct. I'd like to compute a distribution $p'_{1 \dots K}$ that approximates $p_{1 \dots K}$ in some sense while having entropy approximately equal to a given value $H'$.

What algorithm can I use to compute $p'_{1 \dots K}$? Ideally, I'd like a smooth function that can be computed without numerical optimization, since I need to use this in the forward pass of a neural network. I'm flexible about the approximation criterion, as long as it (usually) keeps the rank of the probabilities $p_i > p_j \Longleftrightarrow p'_i > p'_j$.

EDIT:

Note that this computation should be performed at test time. I'm not looking for a training time entropy penalty on an output distribution, but to a method to project an arbitrary distribution to a subpace of the probability simplex with an arbitrary given entropy.

$\endgroup$
1
  • $\begingroup$ Hi Antonio, was I able to answer your question? If so, then please consider upvoting and/or accepting it. Otherwise, what can be clarified? $\endgroup$ May 18 at 4:57
3
$\begingroup$

EDIT: Incorporating my discussion with Antonio, here's one option to adjust a distribution's entropy while preserving the ranks of the values.

Without going into the justification as adjusting a distribution's temperature in statistical mechanics, the short version is this: raise every probability to the same power, then renormalize.

Define $p'_i$ to be $p_i^\beta \over Z$ for some scalar variable $\beta$ that we will solve for. (The normalizing constant $Z = \sum_i p_i^\beta$ ensures that the new distribution is normalized.) Given a sought value $H'$ of the entropy of the distribution $p'$, we can solve the following equation to get our new probability distribution.

$$H' = \sum_{i}p_i' \log p_i' = \frac{1}{Z}\sum_{i}p_i^\beta[\beta \log p_i - \log Z] = \frac{1}{Z}\sum_{i}\left[p_i^\beta\beta \log p_i\right] - \log Z$$

For $\beta > 0$, this will preserve ranks as requested:

$$p_i > p_j \Longleftrightarrow p'_i > p'_j$$

If $\beta = 0$, then you've attained a uniform distribution. As $\beta \to \infty$, you reach a trivial distribution with all mass concentrated on the highest-probability class.


You're describing a constrained optimization problem: you want to learn a distribution $p'$ that performs well, subject to the constraint that is entropy $H(p')$ is (approximately) equal to your given value $H'$.

You can incorporate this as a soft constraint into your objective function. Let's call your network's original objective $J(p, p')$—perhaps this is the cross-entropy.

Incorporate a penalty for deviating from $H'$. Choose a constant $\lambda$ that decides how heavily you want to penalize this deviation. Now optimize this penalized objective instead of the original objective for your network. Note that if $J$ is differentiable with respect to $p'$, then the whole function is differentiable with respect to $p'$.

$$J'(p, p', H') \triangleq J(p, p') + \lambda\left(H' - H(p')\right)^2$$

You can interpret this as a relaxation of a constrained optimization problem. Rather than requiring that the entropy $H(p')$ exactly equals your desired value $H'$, we merely encourage the two values to be similar—their squared difference must be small. This unconstrained problem is much simpler to optimize.

$\endgroup$
5
  • $\begingroup$ I need to control the entropy at test time, I can't rely on training. $\endgroup$ Apr 26 at 22:08
  • 1
    $\begingroup$ Please edit crucial details like that into your question. The entropy at test time can be controlled, but the distribution may not be well-calibrated if you do that. Are you looking for a transformation of a distribution to adjust its entropy? $\endgroup$ Apr 26 at 22:38
  • $\begingroup$ Yes, I'm looking for a transformation of a distribution. $\endgroup$ Apr 26 at 22:48
  • $\begingroup$ Ah, gotcha. A simple option is to adjust the distribution's temperature to match the desired entropy. Given an initial distribution and a desired entropy, this is a single-variable algebra problem, which is straightforward to solve. It'll also preserve the ranks, as you asked originally. But you may not be pleased with the actual values. $\endgroup$ Apr 26 at 22:51
  • $\begingroup$ Updated with details. $\endgroup$ Apr 26 at 23:01

This site is temporarily in read only mode and not accepting new answers.

Not the answer you're looking for? Browse other questions tagged .