Do we have to define a explicit terminal state in reinforcement learning? More specifically,
I think terminal state is necessary for some problem such as travel in the maze etc, however, for some problem where you just want to get the most cumulative reward during any episode, maybe we don not have to have an explicit terminal state.
 A: Continuous problems
In continuous problems, there are no episodes, and no need to worry about handling terminal state at all.
It may be that you have a continuous problem, but you want to limit amount of time spent exploring certain areas of it. It is fine to stop a simulation and start again in order to do this (and clearly you have to stop at least once to end the training). However, in that case, you should avoid calling the stop point a "terminal state" or the experience from start to end "episodes", because these have a specific formalised meaning in Reinforcement Learning. You might call a series of connected state/action/reward data points a "trajectory" or "sample" or something else.
Episodic problems
In episodic problems, an episode must terminate. However, you do not need an explicit terminal state, as in a single state representation that is always the same that somehow the previous state transitions to. Instead you can consider any number of states to be terminal, for any reason that makes sense for your problem. For a state to be considered terminal, you should ensure a few basic things:


*

*It is possible to determine that a state is terminal by inspecting it. 

*Any state value or action value calculated from a terminal state should be zero.

*Any simulation or evaluation of a learning agent should stop once the state is terminal.

*You should not impose termination of an episode based on data that the agent cannot sense or infer. However, you can stop part way through an episode - without terminating it - in order to learn by bootstrapping from value functions (as in TD learning). Whether this works well will depend on the reward structure. For problems like classic board games where the meaningful reward is only on termination of the episode with a win or loss, then stopping prematurely will not work very well.

for some problem where you just want to get the most cumulative reward during any episode

In order for the agent to optimise such a problem, it will need to know when the episode will end. For example, in a simple grid world, the agent might move around visiting squares, some of which grant a reward. If the agent is one step away from a +1 reward, and two steps away in the other direction from a +5 reward, which should it head towards? The optimal decision will depend on remaining time:


*

*If there is only 1 step remaining, then the optimal choice is to head towards the +1 reward. 

*If there are 2 to 4 steps remaining, the agent should head towards the +5 reward

*If there are 5 or more steps remaining, or the agent is in control of whether the episode ends, it does not matter which reward is chosen first (unless you want to prioritise immediate rewards using a discount factor).

*If there are are 5 or more steps remaining and the episode might end randomly at any time, then the agent needs to take into account the expectation that it could reach either rewarding square in the number of steps it will take to get there. High probabilities of ending will favour the +1 square, low probabilities will favour the +5 square (for a fixed probability $p$ of ending after each step, if $5(1-p) \lt 1$ then the +1 square is favourable).
The only way the agent can make an optimal decision in this scenario is if it has the information on how many steps remain (or what the probability is that the episode might end - which it may gain an estimate of through experience). Which in turn means there has to be some data in the state that represents this.
You might "get away with it"
If for some reason you really have an episodic problem, but adding handling for terminal states is an issue, you might still get close enough to optimal policy without it, that the difference is not important.
This is more likely, the more your set up is like a continuous problem. So if you have one or more of:


*

*Long episodes

*Rewards occurring mid-episode, and no special reward associate with ending the episode

*A preference for immediate rewards (expressed via a discount factor $\gamma$)

*State progression tends to cycle through same or similar states, or the highest rewards are gained by maintaining some kind of homeostasis (think of Cart Pole problem - where being balanced is rewarded)
Then it may not matter so much that you don't implement 100% formally correct MDP. The resulting agent may still perform well enough - the most likely flaw is similar to the example above where the agent, unaware of impending end of episode, fails to prioritise actions correctly at the end. For some problems this will little difference to final outcome.
In addition, if your problem is complex enough to require function approximation (such as a neural network), then you cannot guarantee a fully optimal solution anyway. Ignoring proper definitions of terminal state might have a low enough impact that it will not matter.
