some lectures/examples on Q-learning using feature/linear approximation

Question

Most of the resources that I find in the internet deal with Q-learning or SARSA in the discrete sense, when state spaces and actions are finite. There are variety of examples and tutorials for this case.

I am looking for the case when state space is infinite. In this case, it is impossible to list all the states. Hence, an important technique would be to make a features-based representation. However, the resources are few and far in between.

If you know of resources, websites, examples, lectures, even video lectures, especially on the Q-learning/SARSA with features-based representation, it be great to have a look.

Answer 1

Georgia Tech has recently released a reinforcement learning course that covers this in a section called "Generalization." See also Gordon's paper "Stable function approximation in dynamic programming", and Ladoukis & Parr's paper on least-squares policy iteration.

You may also find value in the BURLAP continuous-domain tutorials. (BURLAP is a Java library "for the use and development of single or multi-agent planning and learning algorithms and domains to accompany them.") In particular, the lunar lander example uses SARSA($\lambda$).