The actor-critic model is used within temporal difference learning, which is a method within reinforcement learning, to optimize a process on a state-by-state basis by using the difference between performance and expected performance for each respective state. In other words, there is a set of possible input patterns to the process, each of which constitutes a state (the state in which the agent finds itself). For each state, the actor computes an output, which is evaluated by a performance metric (which can theoretically define a different target output for each state) that returns the reward (performance) signal. Meanwhile, for each state, the critic tracks the expected reward, which is a function of the past reward signals for that state. At each iteration, the difference between the reward signal and the expected reward signal is taken and used to update both the actor's policy (hopefully up a performance gradient) and the critic's performance expectation for that state.
Hypothetically, for some problem there exists two or more states (input patterns) that are similar and where the information that differs between the two is irrelevant. In other words, the target output for the states is the same and the input is similar. It would then be potentially useful for the critic to have a method for recognizing that all states that share a certain similarity are effectively the same state, so that the set of those states can share the same predicted outcome. In particular, if you are dealing with a system that features a massive amount of inputs, and if most of those are non-pertinent at any given point in time, then you don't want to relearn the expected performance for each possible combination of non-pertinent inputs.
My question, then, is how you do this. How do you get a critic to organize different states into state classes, and in a way that improves performance over the long run?