Hidden Markov model with reward

Question

I am looking for a Hidden Markov model that incorporates rewards, i.e., in which the transition between states is dependent on the feedback from the environment (reward). For instance, it could be that there is a higher probability to stay in the same state given we had a reward in the previous play.

For motivation consider the following scenario: A rat is exposed to several combinations of cues (Colors and Odors) attached to different doors. In each trial, it should select the correct door to get some reward. The rat may consider different strategies (Color1, Color2, Odor1, Odor2, Random, etc..) at each trial. Given the behavior of the rat we would like to estimate the underlying strategy of the rat in each trial.

Note that each door has a combination of several cues (e.g., Door1: odor1+color2 and Door2: odor2+color1), thus the estimation the current strategy is not trivial.

Answer 1

Your looking for partially observable Markov decision processes (POMDP).

A POMDP is a formalization of a

• probabilistic (state transition is probabilistic, and depends on prior latent state and chosen action),
• state-discrete (latent state, actions, observation variable are discrete),
• time-discrete,
• sequential (there are multiple decisions in sequence),
• decision problem (the problem is to choose an action to maximize a reward).

Parameter learning

Most of the literature about POMDPs is about acting/planning. When trying to learn parameters for a POMDP, you have to distinguish between many different cases:

• Do you only have observational data (traces of actions, observations, immediate reward)?
• If you can interact with the environment during learning (like in Bayes active learning), you should look into (partially observable) model-free reinforcement learning. There are various approaches for learning POMDP parameters depending on what is given in advance.

If you can interact during learning, then [1] assumes only a given reward function. You can also try searching literature in the field of neuroscience for reinforcement learning models, as your application seems to be closely related.

• [1] Bayes-adaptive POMDPs, Ross, Stephane and Chaib-draa, Brahim and Pineau, Joelle