In following references
 V. Mnih et al., "Recurrent Models of Visual Attention", NIPS, 2014  http://torch.ch/blog/2015/09/21/rmva.html  M. Ranzato et al., "Sequence level training with Recurrent Neural Network", ICLR, 2016
they applied REINFORCE algorithm to train RNN.
To reduce variance of the gradient, they subtract 'baseline' from sum of future rewards for all time steps. According to Appendix A-2 of
. W. Zaremba et al., "Reinforcement Learning Neural Turing Machines", arXiv, 2016
this baseline is chosen as expected future reward given previous states/actions.
My question is training method to get 'baseline'. They train 'baseline' at each time by linear regression (i.e. objective = Mean Square Error) which takes hidden state of RNN as input. How come this training method make sense? Is there anyone to provide relevant reference?
I really appreciate for your comments.