I got confused after reviewing several Q/A on this topic.
As in "how to make a reward function in reinforcement learning", the answer states "For the case of a continuous state space, if you want an agent to learn easily, the reward function should be continuous and differentiable"
While in "Is reward function needed to be continuous in deep reinforcement learning", the answer clearly state "No, there is no requirement for reward to be drawn from any continuous function. That is because the value of Rt is produced by the environment, independently of the parameters θ that the policy gradient is with respect to."
As also discussed in many other papers, blogs etc, reward function selection would paly a big role in convergence. But I'm not sure which of the above statements are more accurate or both of them are correct but they are talking about different aspects?