1
$\begingroup$

The actor-critic model is used within temporal difference learning, which is a method within reinforcement learning, to optimize a process on a state-by-state basis by using the difference between performance and expected performance for each respective state. In other words, there is a set of possible input patterns to the process, each of which constitutes a state (the state in which the agent finds itself). For each state, the actor computes an output, which is evaluated by a performance metric (which can theoretically define a different target output for each state) that returns the reward (performance) signal. Meanwhile, for each state, the critic tracks the expected reward, which is a function of the past reward signals for that state. At each iteration, the difference between the reward signal and the expected reward signal is taken and used to update both the actor's policy (hopefully up a performance gradient) and the critic's performance expectation for that state.

Hypothetically, for some problem there exists two or more states (input patterns) that are similar and where the information that differs between the two is irrelevant. In other words, the target output for the states is the same and the input is similar. It would then be potentially useful for the critic to have a method for recognizing that all states that share a certain similarity are effectively the same state, so that the set of those states can share the same predicted outcome. In particular, if you are dealing with a system that features a massive amount of inputs, and if most of those are non-pertinent at any given point in time, then you don't want to relearn the expected performance for each possible combination of non-pertinent inputs.

My question, then, is how you do this. How do you get a critic to organize different states into state classes, and in a way that improves performance over the long run?

$\endgroup$
1
$\begingroup$

Within the value-function approximation model (which you've adopted?), you can opt to use a compressed encoding of the state-action space instead of a tessellation. Neural networks are sometimes used for this purpose.

What you might try with the critic is to use a supervised clustering algorithm over all states which were recently fed to the actor with the actions as the classes. You could then plunk a new state-action pair into the clustering model. If it flops down in the middle of a pure cluster, and is of the wrong class, it's worth training on. If it's of the right class, you can probably skip it. If it's in an impure cluster, or significantly outside of any dense cluster, probably keep it.

$\endgroup$
  • $\begingroup$ the actor in my system is an NN, so I think I know what you mean by compressed encoding: basically mapping multiple states to the same (or part of the same) action. I'm not familiar with supervised clustering algorithms though. Could you elaborate a bit? If I understand, some algorithm derives state classes, and then any new state that appears to be of an existing class is tested, and if it fails the test or didn't appear to meet any existing class, then it is trained as though belonging to an unknown class? $\endgroup$ – Matt Munson Sep 29 '11 at 23:47
  • $\begingroup$ Yep, that's about right. Thinking about it, my terminology might be a bit off. What you do is run an unsupervised clustering algorithm (e.g. k-means) over all previously evaluated points in the state (not state-action) space. You'll get out a bunch of clusters (k). Test the clusters for purity using the action of each point as a label. Take your new point, drop it into the state space, and determine which cluster it belongs to. If the cluster is highly pure, and the new point's action matches the cluster label, don't bother with it. Otherwise, it's worth learning. $\endgroup$ – John Doucette Sep 30 '11 at 19:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.