I am trying to implement Soft Actor-Critic with automatic Entropy tuning.
The original paper derives the optimal entropy temperature $\alpha$ as follows:
$$ \alpha ^* _t = \arg\min_{\alpha_t} \mathbb{E}_{\mathbb{a}_t \sim \pi^* _t} [-\alpha _t \log \pi^* _t (\mathbb{a}_t | \mathbb{s}_t ; \alpha_t ) - \alpha_t \bar{\mathcal{H}}] $$
In practice, they use the following loss function to update $\alpha$ via stochastic gradient descent
$$ J(\alpha) = \mathbb{E}_{\mathbb{a}_t \sim \pi _t} [-\alpha \log \pi _t (\mathbb{a}_t | \mathbb{s}_t ) - \alpha_t \bar{\mathcal{H}}] $$
However, I have noticed that many implementations of SAC actually use $\log \alpha$ instead of $\alpha$; in other words,
$$ J(\alpha) = \mathbb{E}_{\mathbb{a}_t \sim \pi _t} [-\log \alpha (\log \pi _t (\mathbb{a}_t | \mathbb{s}_t ) + \bar{\mathcal{H}})] $$
I do get that the optimal $\alpha$ will also satisfy:
$$ \alpha ^* _t = \arg\min_{\alpha_t} \mathbb{E}_{\mathbb{a}_t \sim \pi^* _t} [-\log \alpha _t (\log \pi^* _t (\mathbb{a}_t | \mathbb{s}_t ; \alpha_t ) + \bar{\mathcal{H}})] $$
since $\log$ function is monotonic increasing function and $\alpha$ should be positive.
However, I am not very convinced this is still valid under SGD setting. Especially since $\alpha$ decreases during training, I am concerned that the gradient of the loss might be larger. Also, what is the advantage of this modification?
