Can anyone point me in the direction of a formal proof of convergence for a (on/off policy) Monte Carlo control algorithm with (non-)linear function approximation?
In http://incompleteideas.net/book/bookdraft2017nov5.pdf, Section 11.5, the following is stated, but I have not been able to find any references
Among the algorithms investigated so far in this book, only the Monte Carlo methods are true SGD methods. These methods converge robustly under both on-policy and off-policy training as well as for general non-linear (differentiable) function approximators
On http://www.cs.cmu.edu/afs/cs.cmu.edu/project/learn-43/lib/photoz/.g/web/glossary/converge.html different results are presented, where I think the MC case counts as 'residual' (thus converging) naturally since the MC target does not depend on the parameters, however also here I do not find references.
I am a bit confused to find the slide 'Convergence of control algorithms' in http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching_files/FA.pdf, which states MC control does not converge for the non-linear case, and only 'chatters around near-optimal values' for the linear case.
Also papers I have been able to find either consider the prediction case or TD-learning. Am I missing something or is it somehow a trivial extension of the prediction case? Are the sources I mentioned contradicting and if so, why?
