6
$\begingroup$

How do you apply a linear function approximation algorithm to a reinforcement learning problem that needs to recommend an action A in a specific state S?

I've read over a few sources, including this and a chapter in Sutton and Barto's book on RL, but I'm having trouble understanding it. I understand how Q-learning and SARSA work with a normal lookup table by storing expected reward values for (state, action) tuples. And I understand how a vector of parameters can be updated with a reward signal for an LFA.

What I don't understand is where the action comes in when querying and updating an LFA. Both the scholarpedia and S&B don't make any mention of the action when updating LFA weights, so how do they take the action into account? Does an LFA only estimated the value of a state, requiring you to maintain separate LFA calculations for each action?

$\endgroup$

1 Answer 1

5
$\begingroup$

If you haven't yet, check out this page which covers SARSA with LFA: http://artint.info/html/ArtInt_272.html

Sutton's book is really confusing in how they describe how to set up your feature space F(s,a), but in the web page above, they describe it in a simple example. Applying the architecture of theta and F(s,a) from that page to Sutton's algorithm works very well.

Suppose you have 4 possible actions in a state. Create a reward Q distribution (in this case a 4-value array), with one value for each possible action in the given state. Iterate over each action, and for that action, populate the feature space based on what that action will do to/for the agent.

For example, if the agent is directly below a wall, and the chosen action is 'up', there should be a 1 for the feature 'is the agent about to try to move into a wall'. Likewise, for action='right' and wall to the right, the same feature would be a 1, etc. for all other possibilities.

You've probably moved past this problem a while ago, but if not, hope this helped!

$\endgroup$
1
  • $\begingroup$ Yeah, I had already read that link, and it was very helpful. Using that I was able to write my own simple Python implementation. $\endgroup$
    – Cerin
    Oct 21, 2015 at 15:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.