Hidden Markov model with reward

I am looking for a Hidden Markov model that incorporates rewards, i.e., in which the transition between states is dependent on the feedback from the environment (reward). For instance, it could be that there is a higher probability to stay in the same state given we had a reward in the previous play.

For motivation consider the following scenario: A rat is exposed to several combinations of cues (Colors and Odors) attached to different doors. In each trial, it should select the correct door to get some reward. The rat may consider different strategies (Color1, Color2, Odor1, Odor2, Random, etc..) at each trial. Given the behavior of the rat we would like to estimate the underlying strategy of the rat in each trial.

Note that each door has a combination of several cues (e.g., Door1: odor1+color2 and Door2: odor2+color1), thus the estimation the current strategy is not trivial.

Your looking for partially observable Markov decision processes (POMDP).

A POMDP is a formalization of a

• probabilistic (state transition is probabilistic, and depends on prior latent state and chosen action),
• state-discrete (latent state, actions, observation variable are discrete),
• time-discrete,
• sequential (there are multiple decisions in sequence),
• decision problem (the problem is to choose an action to maximize a reward).

Parameter learning

Most of the literature about POMDPs is about acting/planning. When trying to learn parameters for a POMDP, you have to distinguish between many different cases:

• Do you only have observational data (traces of actions, observations, immediate reward)?
• If you can interact with the environment during learning (like in Bayes active learning), you should look into (partially observable) model-free reinforcement learning. There are various approaches for learning POMDP parameters depending on what is given in advance.

If you can interact during learning, then [1] assumes only a given reward function. You can also try searching literature in the field of neuroscience for reinforcement learning models, as your application seems to be closely related.

• [1] Bayes-adaptive POMDPs, Ross, Stephane and Chaib-draa, Brahim and Pineau, Joelle
• Thanks for your answer, I was not aware of this model. So, assuming that the agent is behaving according to the POMDP model to miximize the reward it receives given a partially observed environment (and not simply according to a Markov process), is there an algorithm for estimating the parameters of the model? In other words, is there a "Baum-Welch" like algorithm for POMDP?
– Goek
Jul 4, 2016 at 9:44
• @Goek I can't provide you with a definitive answer. See my update. Jul 4, 2016 at 12:43
• @ziggystart: your answer definitely takes me one step closer. I will check out your directions. Thank you so much for your help.
– Goek
Jul 4, 2016 at 14:17
• I have posted a follow up question here: stats.stackexchange.com/questions/223893/…
– Goek
Jul 19, 2016 at 7:45