The reward range for my current environment is $[-10000, 5000]$, so currently I normalize the reward to $[-2, 1]$. Should I also change the learning rate?
It seems like if you normalize the advantages to have mean of 0 and a standard deviation of 1, it effectively rescales the learning rate by a factor of 1/σ, where σ is the standard deviation of the empirical advantages.
So, if you want the learning rate to stay the same, you should indeed multiply it by σ.
Take a look in 4.1.2 here: http://rail.eecs.berkeley.edu/deeprlcourse-fa17/f17docs/hw2_final.pdf