Understanding policy gradient theorem - What does it mean to take gradients of reward wrt policy parameters?

Question

I am looking for a little clarity on what the policy gradient theorem means. My confusion lies in the fact that the reward $R$ in reinforcement learning is non-differentiable in the policy parameters. As that is the case how does the central objective of policy gradients, finding the gradients of Reward $R$ wrt the parameters of policy function even make sense?

Answer 1

Well first, you're trying to take the gradient of the return, not the reward. Also unless both the environment and the policy is deterministic, you'd be taking the gradient of the expected return. Now for your main question, I'll focus on the discrete state-action space case, where as long as your policy varies smoothly wrt the parameters, there's no reason why the expected return wouldn't be differentiable wrt the parameter. Just because the gradient can't be computed using backpropagation doesn't mean the gradient doesn't exist.

Answer 2

R is a constant used to scale the gradient. Instead of reward it could be returns, advantage, etc.

The gradient with respect to the parameters is found from the log probability of taking a specific action given the current state