I am looking for a little clarity on what the policy gradient theorem means. My confusion lies in the fact that the reward $R$ in reinforcement learning is non-differentiable in the policy parameters. As that is the case how does the central objective of policy gradients, finding the gradients of Reward $R$ wrt the parameters of policy function even make sense?
Well first, you're trying to take the gradient of the return, not the reward. Also unless both the environment and the policy is deterministic, you'd be taking the gradient of the expected return. Now for your main question, I'll focus on the discrete state-action space case, where as long as your policy varies smoothly wrt the parameters, there's no reason why the expected return wouldn't be differentiable wrt the parameter. Just because the gradient can't be computed using backpropagation doesn't mean the gradient doesn't exist.