5
$\begingroup$

I'm looking at sampling functions in the context of reinforcement learning; specifically the explore/exploit problem. A method I've seen pretty often is to derive the action by assigning a score to them and then transforming that score into a probability. One way to do that is by applying the softmax function to the set of scored actions. Then one can sample from those actions with the assigned probabilities. Meaning an action with a high score has a high probability.

What is the relationship between this and Gibbs sampling / Blotzmann sampling? In this paper it is called "Boltzmann exploration", ubc.ca ai book and this suggests that they are pretty similar.

$\endgroup$

3 Answers 3

4
$\begingroup$

Different feedback signals and loss functions

The difference lies in the interpretation of the values / logits. More precisely, how the values / logits are tied to different feedback signals.

First, their similarity

First, let's paraphrase the question. Let $\mathbf{z}\in\mathbb{R}^n$ be proper logits and let $\mathbb{q}\in\mathbb{R}^n$ be (temperature-scaled) values. Then, from their softmax's $$ p_i\ =\ \frac{e^{z_i}}{\sum_je^{z_j}}\ , \qquad \tilde{p}_i\ =\ \frac{e^{q_i}}{\sum_je^{q_j}}\ . $$ it looks like $\mathbf{p}$ and $\tilde{\mathbf{p}}$ are pretty much the same. For instance, both $\mathbf{p}$ and $\tilde{\mathbf{p}}$ live in the probability simplex $\Delta^n=\{x\in[0,1]^n\,|\,\sum_ix_i=1\}$.

Now suppose that $\mathbf{z}\in\mathbb{R}^n$ and $\mathbb{q}\in\mathbb{R}^n$ are outputs of some neural net. In order to learn these quantities, you need to tie them to some sort of feedback signal. This is where they differ.

Categorical signal

A proper logit is usually tied to some MLE objective associated with a categorical distribution, e.g. tensorflow's softmax_cross_entropy_with_logits. $$ \text{loss}\ =\ -\sum_iy_i\,\ln p_i $$ where $\mathbf{y}$ is a one-hot encoded categorical variate. Choosing an objective like this gives $\mathbf{z}$ the interpretation of proper logits.

Gaussian signal

In contrast, the values $\mathbb{q}$ are tied to an MLE objective associated with (multi-variate) Gaussian distribution, i.e. mean squared-error loss. $$ \text{loss}\ =\ (y_i - \tau\,q_i)^2 $$ where now $\mathbf{y}$ is just a real-valued vector in $\mathbb{R}^n$ and $\tau>0$ is the Boltzmann temperature.

Conclusion

Thus, $\mathbf{z}$ and $\mathbf{q}$ differ because they are tied to completely different feedback signals. Applying the same softmax operation to both doesn't undo their differences.

Finally, it should be noted that there is in fact a close relation between the interpretations of $\mathbf{z}$ and $\mathbf{q}$ in the context of reinforcement learning, see [arXiv:1704.06440]. The relation is subtle, but it requires only a small amount of additional structure to derive.

Some practical considerations

The reason why all of this theoretical stuff matters is that in practice the values $\mathbf{q}$ may really not be suited to be interpreted as logits. The problem might be that the values fluctuate too much (resulting in insufficient exploration) or the values be too similar (resulting in too much exploration). In most cases, however, this may be fixed by tuning your Boltzmann temperature $\tau$ to suit your specific environment.

$\endgroup$
2
$\begingroup$

What is the relationship between this and Gibbs sampling / Blotzmann sampling?

Mathematically, the two functions are very similar. Gibbs sampling adds a scaling "temperature" factor which is applied to scores before using them in the softmax.

The scenarios in which they are used are different:

  • Softmax probabilities are used when the function's sole purpose is to generate probabilities, and you are free to adjust the input preferences (or logits) in order to converge on a target distribution. This is the case for policy functions in policy gradient methods.

  • Gibbs sampling can be used when the inputs already represent some other relevant score function (e.g. an action value in reinforcement learning). The temperature parameter gives you some control over the impact in differences of that score between options, but not full control because the scores are measuring something else. This can still be useful for generating policies - both on-policy and a behaviour in off-policy - and has some nice properties for online learning in real systems (it quickly learns to avoid very bad action choices for example), although adding a new important hyperparameter in the form of the temperature value is not great.

$\endgroup$
0
$\begingroup$

The Boltzmann distribution is also known as the Gibbs distribution. Probabilities of states in this distribution have the same form as softmax.

https://en.wikipedia.org/wiki/Boltzmann_distribution

$\endgroup$
2
  • $\begingroup$ While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. - From Review $\endgroup$
    – Antoine
    Sep 7, 2023 at 10:32
  • $\begingroup$ Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. $\endgroup$
    – Community Bot
    Sep 7, 2023 at 11:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.