Controlling the entropy of a distribution What is the best way to control the entropy of a categorical probability distribution?
I have a categorical probability distribution $p_1, p_2, \dots, p_K$, for $K$ small-ish ($< 1000$). Assume that the probabilities are generally distinct. I'd like to compute a distribution $p'_{1 \dots K}$ that approximates $p_{1 \dots K}$ in some sense while having entropy approximately equal to a given value $H'$.
What algorithm can I use to compute $p'_{1 \dots K}$? Ideally, I'd like a smooth function that can be computed without numerical optimization, since I need to use this in the forward pass of a neural network. I'm flexible about the approximation criterion, as long as it (usually) keeps the rank of the probabilities $p_i > p_j \Longleftrightarrow p'_i > p'_j$.
EDIT:
Note that this computation should be performed at test time. I'm not looking for a training time entropy penalty on an output distribution, but to a method to  project an arbitrary distribution to a subpace of the probability simplex with an arbitrary given entropy.
 A: EDIT: Incorporating my discussion with Antonio, here's one option to adjust a distribution's entropy while preserving the ranks of the values.
Without going into the justification as adjusting a distribution's temperature in statistical mechanics, the short version is this: raise every probability to the same power, then renormalize.
Define $p'_i$ to be $p_i^\beta \over Z$ for some scalar variable $\beta$ that we will solve for. (The normalizing constant $Z = \sum_i p_i^\beta$ ensures that the new distribution is normalized.) Given a sought value $H'$ of the entropy of the distribution $p'$, we can solve the following equation to get our new probability distribution.
$$H' = \sum_{i}p_i' \log p_i' = \frac{1}{Z}\sum_{i}p_i^\beta[\beta \log p_i  - \log Z] = \frac{1}{Z}\sum_{i}\left[p_i^\beta\beta \log p_i\right] - \log Z$$
For $\beta > 0$, this will preserve ranks as requested:
$$p_i > p_j \Longleftrightarrow p'_i > p'_j$$
If $\beta = 0$, then you've attained a uniform distribution. As $\beta \to \infty$, you reach a trivial distribution with all mass concentrated on the highest-probability class.

You're describing a constrained optimization problem: you want to learn a distribution $p'$ that performs well, subject to the constraint that is entropy $H(p')$ is (approximately) equal to your given value $H'$.
You can incorporate this as a soft constraint into your objective function. Let's call your network's original objective $J(p, p')$—perhaps this is the cross-entropy.
Incorporate a penalty for deviating from $H'$. Choose a constant $\lambda$ that decides how heavily you want to penalize this deviation. Now optimize this penalized objective instead of the original objective for your network. Note that if $J$ is differentiable with respect to $p'$, then the whole function is differentiable with respect to $p'$.
$$J'(p, p', H') \triangleq J(p, p') + \lambda\left(H' - H(p')\right)^2$$
You can interpret this as a relaxation of a constrained optimization problem. Rather than requiring that the entropy $H(p')$ exactly equals your desired value $H'$, we merely encourage the two values to be similar—their squared difference must be small. This unconstrained problem is much simpler to optimize.
