# Controlling the entropy of a distribution

What is the best way to control the entropy of a categorical probability distribution?

I have a categorical probability distribution $$p_1, p_2, \dots, p_K$$, for $$K$$ small-ish ($$< 1000$$). Assume that the probabilities are generally distinct. I'd like to compute a distribution $$p'_{1 \dots K}$$ that approximates $$p_{1 \dots K}$$ in some sense while having entropy approximately equal to a given value $$H'$$.

What algorithm can I use to compute $$p'_{1 \dots K}$$? Ideally, I'd like a smooth function that can be computed without numerical optimization, since I need to use this in the forward pass of a neural network. I'm flexible about the approximation criterion, as long as it (usually) keeps the rank of the probabilities $$p_i > p_j \Longleftrightarrow p'_i > p'_j$$.

EDIT:

Note that this computation should be performed at test time. I'm not looking for a training time entropy penalty on an output distribution, but to a method to project an arbitrary distribution to a subpace of the probability simplex with an arbitrary given entropy.

• Hi Antonio, was I able to answer your question? If so, then please consider upvoting and/or accepting it. Otherwise, what can be clarified? May 18 at 4:57

EDIT: Incorporating my discussion with Antonio, here's one option to adjust a distribution's entropy while preserving the ranks of the values.

Without going into the justification as adjusting a distribution's temperature in statistical mechanics, the short version is this: raise every probability to the same power, then renormalize.

Define $$p'_i$$ to be $$p_i^\beta \over Z$$ for some scalar variable $$\beta$$ that we will solve for. (The normalizing constant $$Z = \sum_i p_i^\beta$$ ensures that the new distribution is normalized.) Given a sought value $$H'$$ of the entropy of the distribution $$p'$$, we can solve the following equation to get our new probability distribution.

$$H' = \sum_{i}p_i' \log p_i' = \frac{1}{Z}\sum_{i}p_i^\beta[\beta \log p_i - \log Z] = \frac{1}{Z}\sum_{i}\left[p_i^\beta\beta \log p_i\right] - \log Z$$

For $$\beta > 0$$, this will preserve ranks as requested:

$$p_i > p_j \Longleftrightarrow p'_i > p'_j$$

If $$\beta = 0$$, then you've attained a uniform distribution. As $$\beta \to \infty$$, you reach a trivial distribution with all mass concentrated on the highest-probability class.

You're describing a constrained optimization problem: you want to learn a distribution $$p'$$ that performs well, subject to the constraint that is entropy $$H(p')$$ is (approximately) equal to your given value $$H'$$.

You can incorporate this as a soft constraint into your objective function. Let's call your network's original objective $$J(p, p')$$—perhaps this is the cross-entropy.

Incorporate a penalty for deviating from $$H'$$. Choose a constant $$\lambda$$ that decides how heavily you want to penalize this deviation. Now optimize this penalized objective instead of the original objective for your network. Note that if $$J$$ is differentiable with respect to $$p'$$, then the whole function is differentiable with respect to $$p'$$.

$$J'(p, p', H') \triangleq J(p, p') + \lambda\left(H' - H(p')\right)^2$$

You can interpret this as a relaxation of a constrained optimization problem. Rather than requiring that the entropy $$H(p')$$ exactly equals your desired value $$H'$$, we merely encourage the two values to be similar—their squared difference must be small. This unconstrained problem is much simpler to optimize.

• I need to control the entropy at test time, I can't rely on training. Apr 26 at 22:08
• Please edit crucial details like that into your question. The entropy at test time can be controlled, but the distribution may not be well-calibrated if you do that. Are you looking for a transformation of a distribution to adjust its entropy? Apr 26 at 22:38
• Yes, I'm looking for a transformation of a distribution. Apr 26 at 22:48
• Ah, gotcha. A simple option is to adjust the distribution's temperature to match the desired entropy. Given an initial distribution and a desired entropy, this is a single-variable algebra problem, which is straightforward to solve. It'll also preserve the ranks, as you asked originally. But you may not be pleased with the actual values. Apr 26 at 22:51
• Updated with details. Apr 26 at 23:01