Add maximum time step to value iteration algorithm What would a value iteration algorithm look like if I specify a maximum time step?
For example, from a given state the environment does not reach a terminating state but instead should terminate because it has exceeded the maximum number of steps (and thus the value for that state would the future return up until the number of steps surpasses n). Essentially, I am trying to learn a truncated policy but I am having difficulty figuring out how to do that using dynamic programming. I suppose another way to phrase this would be depth limited value iteration.
 A: If you add a maximum time step to an environment, then that does not change value calculation algorithms at all. It changes the state representation. You should add either the time step or time remaining as a new property of the state.
Without considering the current time step in the state, if an agent could be in the same state, but with a different amount of time remaining, then it could not calculate a consistent value for expected future reward. The expected future reward will usually be correlated with the amount of remaining time. Without the time in the state, the environment would become non-Markov (or partially observable). Whilst there are some algorithms that may solve this - an RNN variant of DQN springs to mind because it would learn its own time step counter - value iteration is not one of them.
There may be exceptions to this where the state already implies a time step. A good example is Tic Tac Toe (Noughts and Crosses), because the number of filled squares equals the current time step. An environment like this where you added a rule to stop after a certain number of turns (before the board was filled) would not need a separate turn counter.
