In the very simple examples of reinforcement learning (gridworld, mountain car), we use real numbers or some elementary functions as reward functions.

When state spaces become larger and larger, and eventually continuous, the states become harder and computationally expensive to define. So here comes the idea of functional approximation, where we can use features to define.

I have always thought of 'features' (from the word itself) as qualities which I can measure. For example: how far is the agent from a certain obstacle, or how far the agent is from goal position, etc. But I have never seen this in examples/sample codes. In Sutton, there is talk of radial basis functions for features.

here are my questions:

  1. What is the role of radial basis functions in functional approximation?
  2. Can my idea of 'features' (per definition) work?
  3. Do you have some examples in github, or otherwise which shows this implementation?



Your idea of fratures is actually correct, I implemented a RL agent that learns to play Pac-Man: in a small environment it can use classic RL algorithms and actually store a Q table for each possibile state/action tuple, but as you said when the space becomes larger it becomes intractable. In this case, I used features like the presence if ghosts or food nearby as an inout of a neural network, and I was able to train it to win every game. Radial Basis functions allow you to extend, in a sense, your input to a ML algorithm or a NN, if used on the input data as a kernel, to apply the "kernel trick". In this way you can exploit non linear aspects of the i put data to classify with non linear boundaries. A similar concept is thus applied in the field of RL, using the state input data.

  • $\begingroup$ Hi, thanks for your answer. I am not totally clear about how the radial basis functions can be used to work as features. I am not entirely sure how my idea of 'features' (which you said works) and how radial basis functions can work hand in hand. What is their connection? How are these two related? $\endgroup$ – cgo Mar 13 '17 at 6:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.