Association between Current state/action and the far future reward How the Agent make the association between the current $Q_t(s_t,a_t)$ and a far future reward that by nature of my environment we get reward at least after 10-15 time steps from the action taken. If the rewards are collected in Replay Buffer and sampled at random then its probably gonna be broken, else if online training is used what part of the $Q$ update will consider the future reward ?
Edit:
I have noticed another similar question Delayed Rewards in Reinforcement Learning. Yet, but it had fixed delay period, here the delay is stochastic and reward may not occur.
 A: All value-based reinforcement learning (RL) methods use some form of backup between time steps. The backups take values known from later time steps, and use it to calculate values expected at earlier time steps.
This is an important part of RL method theory and design, and there is a visualisation of it called "backup diagrams" which you will find many examples of in the early chapters of Reinforcement Learning: An Introduction.
In Q learning, you can see the relationship between values at $t$ and $t+1$ in the update:
$$Q(s_t, a_t) \leftarrow Q(s_t, a_t) + \alpha(r_{t+1} + \gamma \text{max}_{a'} Q(s_{t+1}, a') - Q(s_t, a_t))$$
That is, the value of $Q(s_t, a_t)$ is being updated, and the values of $r_{t+1}$ and $s_{t+1}$ are used directly in the caclulation of that update. Over many repetitions of this update rule, values are backed up from later time steps (where you have better knowledge of the eventual rewards seen) to earlier time steps.
Specifically for tabular Q learning, you were concerned about initial inaccuracy of the bootstrap estimates. The resulting bias is a concern, and in some off-policy methods it can prevent covergence. However, with reasonable assumptions in tabular Q learning, it can be proved to converge.
