I am still new to Deep-RL. I was reading DeepMind's paper "Playing Atari with Deep Reinforcement learning" and am having trouble understanding how these games can be represented as a finite markov decision process. Don't these games technically have an infinite number of possible states and possible actions?
The paper explains that:
Since the agent only observes images of the current screen, the task is partially observed and many emulator states are perceptually aliased, i.e. it is impossible to fully understand the current situation from only the current screen xt. We therefore consider sequences of actions and observations, st = x1,a1,x2,...,at-1,xt, and learn game strategies that depend upon these sequences. All sequences in the emulator are assumed to terminate in a finite number of time-steps. This formalism gives rise to a large but finite Markov decision process (MDP) in which each sequence is a distinct state. As a result, we can apply standard reinforcement learning methods for MDPs, simply by using complete sequence st, as the state representation at time t.
I fail to see the correlation between the sequences terminating in a finite number of time-steps to the rise of finite MDP. Is it saying that the finite MDP, in this instance, is just an insanely large chain that has "info" on almost all the possible sequences? Therefore, it can make predictions on the best action for any given state that the emulator sees? If possible could you give me an example of games which are continuous state spaces and explain why? Thanks!
DeepMind Paper: Link