# Why $A_{t-1}$ in reinforcement learning history $H_t = O_1, R_1, A_1, ..., A_{t-1}, O_t, R_t$? [closed]

I learn with David Silver's slides reinforcement learning. His definition of the history $$H_t$$ is:

$$H_t = O_1, R_1, A_1, ..., A_{t-1}, O_t, R_t$$

$$O =$$ observations

$$R =$$ rewards

$$A =$$ actions

Why do we have $$A_{t-1}$$ and not $$A_t$$?

• Have you transcribed the start of $H_t$ correctly? I would expect to either see $O_0, A_0, R_1,$ at the start or $O_1, A_1, R_2$ - please could you reference the specific lecture and slide number, because your current link is to a page with many different lectures and many different slides for each lecture, and I don't want to search through them all to discover your reference Commented Mar 17, 2022 at 15:56
• Slide 18. I transcribed the start of $H_t$ correctly. @NeilSlater
– yemy
Commented Mar 17, 2022 at 17:08

David Silver has chosen not to include the action taken at time $$t$$ because the history is being referenced so that the agent can choose $$A_t$$.

The observation $$O_t$$ is available because the agent has just arrived at time step $$t$$ and received the observation. It needs to make a decision on what to do next.

Histories at other stages of the process exist in principle. E.g. just after taking the action, but before receiving the immediate reward and next state is a valid point to construct a history of the trajectory so far. However, it is a less useful history when considering what inputs you have available for a policy function.

• Okay, so it can be written also like this $H_t = O_1, R_1, A_1, ..., A_t, O_{t+1}, R_{t+1}$ if I understand you correctly?
– yemy
Commented Mar 17, 2022 at 17:15
• @MaxHager Yes, but only if you wanted to consider the choice of action at $A_{t+1}$ - it would be more natural to consider action choice at time $t$ then write the history definition to match, than to decide to stop the trajectory at time $t+1$ just before action choice, so that there isn't an awkward-looking $t-1$ in the summary of the trajectory Commented Mar 17, 2022 at 18:58