In the derivation of the Gradient Bandit Algorithm in Chapter 2.8 of the Reinforcement Learning book by Sutton & Barto they introduce a introduce a baseline term $B_t$ and I can't seem to figure out the intuition for adding such a term. I understand how it affects the math and how the equation remains unchanged due to its addition. Is there a mathematical logic that advocates for adding such terms? It seems unintuitive to add any constant parameters arbitrarily
The crucial assumption for baseline term $B_t$ at at each time step here is independence of $x$ as mentioned in your quoted section and you also understood it has no effect to the bandit gradient algo which is an instance of stochastic gradient ascent as proved by your quote. Baseline term is often heuristically introduced in reinforcement learning including REINFORCE with Baseline method of the policy-based gradient methods family to reduce update variance and speed up learning. Actually at the end of your quote the heuristic reason is give as:
The choice of the baseline does not affect the expected update of the algorithm, but it does affect the variance of the update and thus the rate of convergence (as shown, for example, in Figure 2.5). Choosing it as the average of the rewards may not be the very best, but it is simple and works well in practice.
In summary without a good baseline any gradient-based algo's update to the policy parameters are noisy usually resulting in very slow convergence.
