2
$\begingroup$

I am trying to implement Soft Actor-Critic with automatic Entropy tuning.

One thing I noticed is that the authors, and also the majority of implementations of SAC use $-\text{dim}(|\mathcal{A}|)$ as entropy target $\bar{\mathcal{H}}$ in the loss function of $\alpha$, and I am confused where this exact value came from. I have read the answer to this question, but I am still not fully convinced.

I do understand the intuition behind setting $\bar{\mathcal{H}}$ proportional to $\text{dim}(|\mathcal{A}|)$. Let's say, for example, we consider the target distribution of $\text{tanh}(\bf{u})$ to be $\text{Uniform}(-1, 1)$ for each dimension (I am not sure if this is a good assumption though), then the entropy target should be $\text{dim}(|\mathcal{A}|) \ln(2)$. But I am not sure why the majority of the implement uses exactly $-\text{dim}(|\mathcal{A}|)$.

Is there any specific reason for this setting? Or, are there any papers or articles about how the target entropy setting affects the performance of SAC?

Edit:

Just in case, the loss function of $\alpha$ I am referring to is Eq18:

$$ J(\alpha) = \mathbb{E}_{\bf{a}_t \sim \pi_t} [- \alpha \log \pi_t(\bf{a}_t | \bf{s}_t) - \alpha \bar{\mathcal{H}}] $$

$\endgroup$

1 Answer 1

3
$\begingroup$

If you are minimizing a loss function $f$, and there is a function $g$ that is proportional to $f$ something like this
$$ g = k \cdot f $$ then by minimizing $f$ you are also minimizing $g$.

It also extends to all transformations of f that do not change the ordering. If for a transformation $h(f)$ if you were to sort both $f$ and $h(f)$ and the order of the result is identical, then minimizing on $f$ is the same as minimizing on $h(f)$. Its said a little awkwardly, sorry.

Positive examples:

lets say that $f=\operatorname{abs}(x)$, you could minimize $h(f)=f^2$ and get the minimum would be the same.

lets say that $f=\operatorname{abs}(x)$ and $x$ is constrained to $0<x$. you could minimize $h(f)=\log(f)$ and get to the same location.

Negative examples (how to do it wrong):

lets say that $f=\operatorname{abs}(x)$ and $x$ is NOT constrained to $0<x$. you could minimize $h(f)=\log(f)$ and you would run into the discontinuity at $x=0$ and your optimization would fail.

$\endgroup$
3
  • $\begingroup$ Can you elaborate on how this is exactly connected to the selection of entropy target $\hat{\mathcal{H}}$? The explanation is to vague and I really do not see the connection with the loss function in this answer and the loss function used for adjusting $\alpha$. $\endgroup$
    – esh3390
    Commented Jun 1, 2023 at 14:23
  • $\begingroup$ Where it says "then the entropy target should be $dim(|A|)ln(2)$" you are differing about a constant of proportionality. I will check the paper, but some folks maximize the negative loss and others minimize the negative gain. Signs can get lost, but inverting during minimize can be a very bad thing. It might be a typo. Let me check. $\endgroup$ Commented Jun 1, 2023 at 16:28
  • $\begingroup$ The $\text{dim}(|\mathcal{A}|) \ln (2)$ part was just an example that is not even sure if it is valid or not. Also, I wouldn't say the loss function for $\alpha$ used in the paper is proportional to $\bar{\mathcal{H}}$. $\endgroup$
    – esh3390
    Commented Jun 5, 2023 at 5:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.