# Maximizing cross entropy reinforcement learning

I have read that in reinforcement learning, maximizing the entropy enables the policy to behave more randomly. My question comes in three parts:

(1) In the equation below in the cross-entropy term what does the dot • symbol stand for? (2) if maximizing the entropy also makes the policy behave more randomly - then does that mean that it prevents training an optimum policy to convergence?

entropy = -tf.reduce_sum(policy * log_policy, 1, name="entropy")


However the policy is the output of the softmax and not the actual label as is usually the case for cross entropy. Is there a reason why the label (0 for move left, or 1 for move right) was not used.

1. expr$$(\cdot)$$ is shorthand for a function which takes as input $$x$$ and returns expr$$(x)$$. The distinction here is between expr$$(x)$$ -- an output of the function -- and expr$$(\cdot)$$ which is the function itself.
2. This depends on the exact algorithm. An off-policy RL algorithm can converge to the optimal solution even if the rollout policy is not optimal. For on-policy algorithms, $$\alpha$$ is generally small enough that it's not a concern. $$\alpha$$ can also be annealed to 0 over time.
3. Entropy is a functional of probability distributions, such as $$\pi(\cdot|s)$$. A sample from a probability distribution doesn't have entropy.