In exercise 3.6 of the book, 'An Introduction to Reinforcement Learning' by Sutton, R. and Barto, A. They ask the following question at the very end of the chapter 3.5 (which introduces the Markov Property).
Broken Vision System: Imagine that you are a vision system. When you are first turned on for the day, an image floods into your camera. You can see lots of things, but not all things. You can't see objects that are occluded, and of course you can't see objects that are behind you.
i) After seeing that first scene, do you have access to the Markov state of the environment?
ii) Suppose your camera was broken that day and you received no images at all, all day. Would you have access to the Markov state then?
I don't quite understand what they are asking. What is the 'Markov state of the environment'?
What I've thought so far:
i) It doesn't have the Markov property, as what the state of the environment will be in the next image does not depend entirely on what is in the current image (although it may well be a good approximation). I don't quite know what that says about the Markov state of the environment though?
Or is the Markov state of the environment just all the information in the environment at that point in time? In which case it can't see occluded objects and objects not in field of view, so it doesn't have access to all that information (the state).
ii) I think its best I wait to get feedback on the first question before any kind of assumptions about answers to this second question.
Thanks, I'll be very grateful if you help me patch up my understanding of this :)