Can the observation function in a POMDP be a function of the previous state?

Question

I would like to model my problem with a Partially Observable Markov Decision Process (POMDP) but I have as an observation the previous state $o_t = s_{t-1}$.

However, I see in all formal definitions of POMDPs that the observation function can only be a function of the current state, and previous action: $o_t=\mathcal{O}(s_t, a_{t-1})$ and not a function of the previous state.

A imagine a simple way could be to define $\bar{s_t} = (s_t, s_{t-1})$ but I was wondering if there exists some cases where we can have $o_t=\mathcal{O}(s_t, a_{t-1}, s_{t-1})$.

Thank you very much for your help!

Answer 1

In the following I'll switch from your deterministic observation notation $o_t = {\mathcal O}(s_t,a_{t-1})$ to more general probabilistic notation: $${\mathcal O}(o| s' , a) = P\left(O_t = o |S_t = s', A_{t-1} = a \right) = P(o|s',a)$$ $${\mathcal O}(o| s' , a, s) = P\left(O_t = o |S_t = s', A_{t-1} = a, S_{t-1} = s \right) = P(o|s',a,s)$$ Let's start with the Bayesian update to the belief state if $a$ and $o$ are known ($N$ is the normalization factor.): $$b'(s') = \sum_sP\left(s' | o , a, s\right) b(s)= \frac1N\sum_sP\left(o|s',a,s\right) P\left(s'|a,s\right)b(s)$$

In standard case, following the usual definition of the POMDPs, the first probability is independent on the previous state $P\left(o|s',a,s\right) = {\mathcal O}(o| s' , a)$ and we get the standard POMDP belief update expression: $$b'(s') = \frac1N{\mathcal O}(o| s' , a) \sum_s T\left(s'|a,s\right)b(s)$$ Which also can be understood as an application of the forward algorithm from HMM theory.

In your case, the observation probability is not getting out of the summation and the belief update looks like this: $$b'(s') = \frac1N\sum_s {\mathcal O}(o| s' , a, s) T\left(s'|a,s\right)b(s)$$ I could be missing something, but, actually, I can't see much of the drawback for generalizing the definition of POMDPs like that. It looks to me that most of the standard POMDP analysis, including all the $\alpha$-vector business, can be straightforwardly augmented to incorporate the ${\mathcal O}(o| s' , a, s)$ dependence.

My guess as to why the original POMDP definition was constrained like this, was to keep it similar to HMMs. However, we don't need all the fancier (forward-backward, Viterbi) algorithms from the HMM theory. While similar augmentation for the HMM forward algorithm should also work.

Finally. Your case is a bit more constrained that the discussion above - we have perfect observtation of the previous state $s_{t-1}$. So at every point your belief state for the current state $s_t$ is just application of the transition probability: $$b'(s_t) = T(s_t|s_{t-1},a_{t-1})$$ I think you can go through POMDP theory with that modification, but I suspect that the result will be exactly similar to the approach that you are suggesting - just double the state space $\bar{s}_t = (s_t, s_{t-1})$.