1
$\begingroup$

I have read that in reinforcement learning, maximizing the entropy enables the policy to behave more randomly. My question comes in three parts:

(1) In the equation below in the cross-entropy term what does the dot • symbol stand for?

enter image description here

(2) if maximizing the entropy also makes the policy behave more randomly - then does that mean that it prevents training an optimum policy to convergence?

(3) I have seen entropy being implemented as

entropy = -tf.reduce_sum(policy * log_policy, 1, name="entropy")

However the policy is the output of the softmax and not the actual label as is usually the case for cross entropy. Is there a reason why the label (0 for move left, or 1 for move right) was not used.

$\endgroup$
1
$\begingroup$
  1. expr$(\cdot)$ is shorthand for a function which takes as input $x$ and returns expr$(x)$. The distinction here is between expr$(x)$ -- an output of the function -- and expr$(\cdot)$ which is the function itself.
  2. This depends on the exact algorithm. An off-policy RL algorithm can converge to the optimal solution even if the rollout policy is not optimal. For on-policy algorithms, $\alpha$ is generally small enough that it's not a concern. $\alpha$ can also be annealed to 0 over time.
  3. Entropy is a functional of probability distributions, such as $\pi(\cdot|s)$. A sample from a probability distribution doesn't have entropy.
$\endgroup$
1
  • $\begingroup$ I noticed that you are very knowledgeable in reinforcement learning : I made a new post with a different question - I would be grateful if you could look at it : stats.stackexchange.com/questions/369484/… $\endgroup$
    – Mellow
    Sep 30 '18 at 19:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.